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SUMMARY 


Embedded  training  (ET)  is  receiving  increasing  attention  in  the  world  of  military  training. 
Historically,  the  Air  Force  Human  Resources  Laboratory  (AFHRL)  has  devoted  few  resources 
to  embedded  training  research  and  development  (R&D)  issues.  This  study  was  undertaken 
(a)  to  assess  the  extent  to  which  embedded  training  will  be  used  in  tne  Air  Force  in  the 
future,  and  (b)  to  develop  an  R&D  strategy  for  embedded  training.  It  focuses  on  how  AFHRL 
could  use  its  strengths  to  address  the  problems  of  potential  Air  Force  users  of  embedded 
training  R&D.  In  doing  so,  it  attempts  to  apply  private  sector  strategic  planning  methods  to 
a  military  laboratory. 

Embedded  training  is  a  training  capability  which  is  designed  into  or  added  onto  operational 
equipment.  During  the  first  phase  of  this  study,  the  authors  reviewed  the  literature  on  embedded 
training  and  surveyed  military  embedded  training  developers  and  users  to  identify  systems. 
They  established  a  database  of  embedded  training  systems,  from  which  14  of  the  more 
promising  systems  were  selected  for  further  in-depth  study.  Air  Force  program  managers  were 
contacted  to  find  out  what  factors  determined  whether  training  would  be  embedded  into  each 
system.  Phase  I  led  to  three  important  conclusions1 

1.  Embedded  training  is  found  predominately  in  signal  processing  systems  like  radar  and 
sonar. 

2.  For  various  reasons,  embedded  training  is  not  used  for  maintenance  training. 

3.  Embedded  training  is  used  predominately  for  continuation  training. 

During  the  second  phase  of  the  study,  the  authors  visited  four  potential  AFHRL  customers 
to  find  out  what  problems  they  were  encountering  with  embedded  training.  In  addition,  they 
assessed  the  capabilities  of  private  sector  firms  engaged  in  or  capable  of  performing  embedded 
training  research.  The  authors  then  identified  technology  development  areas  that  addressed 
potential  R&D  customer  needs  and  that  built  upon  AFHRL’s  special  strengths.  This  assessment 
resulted  in  a  proposed  embedded  training  R&D  strategy  for  AFHRL 

The  study  identifies  six  embedded  training  technology  development  areas  in  which  AFHRL 
could  conduct  R&D.  Each  area  is  discussed  in  terms  of  the  potential  customer,  relevant 
AFHRL  strengths,  and  risks.  If  AFHRL  decided  to  go  into  embedded  training  R&D,  the  authors 
recommend  that  three  AFHRL  divisions  each  be  given  responsibility  for  marketing  a  technology 
deve'opment  area  to  a  specific  potential  customer: 

Division  ET  Issue  Potential  Customer 


AFHRL/OT 

ET  Safety 

Aeronautical  Systems  Division 

AFHRL7ID 

Embedded  Job 
Performance 

Measures  and 

Intelligent  Tutors 

Standard 

Systems  Center 

AFHRL/LR 

Total  Training 

Systems 

Air  Force 

Space  Commad 

The  authors  recognize  that  if  AFHRL  chooses  to  develop  an  embedded  training  R&D  activity, 
the  Laboratory  would  have  to  invest  significant  time  in  acquiring  the  support  of  potential 
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customers  for  embedded  training  R&D.  So,  if  the  Laboratory  decides  to  proceed,  it  should 
designate  a  single  individual  as  AFHRL's  embedded  training  advocate.  This  individual  would 
be  responsible  for  coordinating  the  Laboratory’s  embedded  training  R&D,  serving  on  the 
Embedded  Training  Working  Group  of  the  Joint  Services  Technical  Coordinating  Group,  and 
spearheading  the  Laboratory’s  marketing  efforts. 
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PREFACE 


This  strategic  planning  study  was  undertaken  by  the  Laboratory  with  the  assistance  of 
Air  Training  Command’s  Training  Technology  Applications  Program  under  contract  No. 
F41689-86-D-0009.  This  is  the  final  report  of  that  study,  which  centered  around  establishing 
a  viable  R&D  strategy  for  the  Laboratory  in  embedded  training. 

The  project  benefited  from  many  helpful  suggestions  from  Drs.  Herbert  J.  Clark  and 
Robert  W.  Stephenson.  The  authors  would  like  to  express  their  appreciation  for  Dr.  Clark’s 
and  Dr.  Stephenson's  interest  and  help. 
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RESEARCH  AND  DEVELOPMENT  STRATEGIES 
FOR  EMBEDDED  TRAINING 

l.  INTRODUCTION 


Reasons  for  an  Embedded  Training  Strategic  Planning  Study 

Several  considerations  have  led  the  Air  Force  Human  Resources  Laboratory  (AFHRL)  to 
undertake  an  embedded  training  study  at  this  time.  First,  weapon  systems  are  becoming  far 
more  complex  and  are  more  likely  involve  computerized  components.  The  computing  power 

of  today's  weapon  and  support  systems  provides  designers  with  an  opportunity  to  build-in 

capabilities  to  he'p  operators  maintain  their  proficiency.  Second,  because  embedded  training 

is  already  being  used  throughout  the  Air  Force  for  certain  kinds  of  training  applications,  it  is 
a  natural  area  for  AFHRL  to  investigate.  Although  embedded  training  is  not  in  extensive  use, 
it  is  a  viable  training  option  and  particularly  well  suited  "continuation  training."  Third,  reductions 
in  Defense  resources  may  lead  the  Air  Force  to  give  embedded  training  opportunities  more 

serious  consideration.  Faced  with  declining  resources,  the  Air  Force  should  examine  embedded 
training  as  an  option  that  might  provide  more  cost-effective  solutions.  In  a  future  scenario 
with  fewer  planes,  fewer  funds  for  operational  deployments  and  sorties,  fewer  replacement 

personnel,  and  less  initial  training,  embedded  training  could  provide  an  attractive  way  for 
sustaining  essential  wartime  skills. 

The  Army  and  the  Navy  have  devoted  more  attention  than  the  Air  Force  to  embedded 

training.  The  Navy  is  conducting  studies  to  determine  the  applicability  of  embedded  training 

to  various  fielded  and  developmental  systems.  These  include  such  applications  as  the 

AN/SPA-25G  radar  system,  the  MK-117  fire  control  system,  the  AN/BSY-2  combat  control  system, 
and  the  OJ-194  Nava)  Tactical  Data  System  (MATRIS  Summary  Report,  1989).  Several  of 
these  systems  are  generally  similar  to  existing  Air  Force  systems  and  demonstrate  the  utility 
of  embedded  training.  The  Army  has  conducted  extensive  research  in  the  area  of  embedded 
training.  The  Army  Research  Institute  (ARI)  has  produced  a  10-volume  report  on  implementing 
embedded  training  (Finley,  Alderman,  Peckham,  &  Strasel,  1988).  This  research  has  indicated 
the  viability  of  embedded  training  as  an  alternative  to  other  forms  of  training. 

Embedded  training  has  not  been  a  major  focus  of  research  at  AFHRL.  The  reasons  stated 
above  have  led  the  Laboratory  to  consider  it  for  inclusion  on  their  research  and  development 
(R&D)  agenda.  The  present  investigation  contributes  to  'hat  goal  in  two  ways:  First,  it 
assesses  the  extent  to  which  embedded  training  will  be  used  in  the  Air  Force  in  the  future; 
second,  it  devises  an  embedded  training  R&D  strategy  for  AFHRL.  The  remainder  of  this 
chapter  will  discuss  what  embedded  training  is  and  will  outline  will  outline  the  approach  used 
in  the  study. 


Definition  of  Embedded  Training 


Embedded  training  (ET)  has  been  defined  by  the  Joint  Technical  Coordinating  Group  for 
Training  Systems  and  Devices  (JTCG-TSD)  as  a  training  capability  which  is  designed  into  or 
added  onto  operational  equipment  (Study  Plan  for  Embedded  Training,  1989).  The  operational 
equipment  itself  thus  becomes  the  delivery  medium  for  the  instruction. 

Embedded  training  can  be  either  fully  incorporated  into  operational  software  (i.e. ,  "built-in"), 
or  "strapped  onto"  the  operational  system.  An  example  of  a  strap-on  embedded  training  system 
is  the  F-16  On-Board  Electronic  Warfare  Simulator  (F-16  OBEWS).  This  system  consists  of 
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a  pod  that  is  attached  to  an  operational  aircraft  to  enable  it  to  "function"  in  the  embedded 
training  mode.  An  example  of  a  built-in  system  is  the  Airborne  Operational  Computer  Program 
(AOCP)  software,  which  is  part  of  the  Airborne  Warning  and  Control  System  (AWACS)  weapons 
direction  system.  The  AOCP  allows  the  operator  to  use  the  AWACS  equipment  in  a  simulation 
mode  to  practice  switch  actions  necessary  to  direct  aircraft  to  targets.  In  either  case,  the 
critical  point  is  that  the  operator's  perception  of  the  difference  between  the  training  mode  and 
the  operational  mode  of  the  system  is  minimal.  The  training  looks  and  feels  "real." 

Although  the  JTCG-TSD  definition  focuses  on  operational  readiness,  it  does  not  rule  out 
other  types  of  training  which  might  be  embedded.  The  following  sections  of  this  chapter  will 
identify  various  kinds  of  embedded  training  and  describe  some  features  of  each. 


Types  of  Embedded  Training 

Simulation  embedded  training  is  the  prototypal  form  of  embedded  training;  that  is,  it  meets 
the  definition  to-the-letter.  It  is  most  common  among  computer  systems  dedicated  to  a  specific 
application.  In  simulation  embedded  training,  some  form  of  job-related  scenario  is  presented 
on  the  displays  and/or  console  normally  used  for  system  operation.  The  F-16  OBEWS  system 
is  a  typical  example  of  this  kind  of  embeddec  training.  The  F-16  OBEWS  system  allows  the 
pilot  to  conduct  electronic  warfare  exercises  against  single  or  multiple  threats  without  other 
aircraft  to  simulate  those  threats.  Although  there  are  no  other  aircraft  involved  in  the  training 
scenario,  the  pilot  sees  the  same  displays  on  his  sensors  as  he  would  if  he  were  flying  an 
actual  combat  mission.  Frequently,  systems  with  simulation  embedded  training  are  strategic 
systems  such  as  the  Ballistic  Missile  Early  Warning  System  (BMEWS)  or  PAVE  PAWS  phased 
array  radar  system.  The  "real-world"  scenarios  found  in  these  systems  are  frequently  used 
to  train  individuals  or  combat  teams  on  those  procedures  which  they  must  follow  during 
operational  situations.  The  simulation  embedded  training  often  provides  practice  (in  the  team 
training  environment)  for  improving  the  operator's  decision  making  skills  The  embedded  training 
features  of  these  systems  allow  the  system  operators  to  see  ballistic  missile  threats  on  their 
sensors  that  they  would  otherwise  not  be  capable  of  experiencing  except  in  an  emergency 
situation. 

The  definition  of  embedded  training  may  be  expanded  to  include  more  than  system-specific 
software.  Embedded  training  can  take  one  of  two  other  forms:  tutorial  or  job  aiding.  In 
tutorial  embedded  training,  screens  of  text,  graphics,  and/or  questions  are  presented  to  th° 
trainee.  This  form  of  embedded  training  is  commonly  used  to  provide  the  operator-trainees 
fully  developed  lessons  on  some  topic  related  to  the  system’s  use  on  the  job.  It  is  particularly 
useful  for  teaching  new  information  or  skills.  Tutorial  embedded  training  differs  from  simulation 
embedded  training  in  that  it  does  not  attempt  to  precisely  emulate  the  actual  job  environment; 
tutorial  embedded  training  emulates  the  classroom  or  study  environment  (i.e. ,  it  is  like  a 
computer-based  training  (CBT)  lesson).  The  World  Wide  Military  Command  and  Control  System 
(WWMCCS)  provides  a  good  example  of  tutorial  embedded  training,  "here  are  tutorial  lessons 
available  on  WWMCCS  for  a  wide  variety  of  applications.  The  operator-trainee  can  select 
those  lessons  which  would  be  most  helpful  for  his  specific  job  assignment. 

In  job  aiding  embedded  training,  information  is  made  available  to  the  system  operator  as 
needed,  in  the  form  of  help  screens,  glossaries,  and  other  such  utilities.  Everyone  who  has 
used  commercial  software  products  which  have  "help"  features  has  experienced  some  type  of 
job  aiding  embedded  training.  Job  aiding  embedded  training  can  be  used  to  help  new  users 
become  oriented  to  a  system  but  seldom  attempts  to  train  beyond  that  level.  Neither  tutorial 
nor  job  aiding  embedded  training  meets  the  highly  realistic  operational  job  practice  capability 
of  simulation  embedded  training. 
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Embedded  training  offers  unique  training  advantages  and  disadvantages.  Because  it  utilizes 
the  actual  operational  equipment,  it  is  usually  readily  available.  There  is  no  requirement  for 
classrooms,  instructors,  simulators  or  other  equipment.  Most  importantly,  simulation  embedded 
training  allows  trainees  to  practice  with  actual  equipment  and  facilitates  the  transfer  of  skills 
to  actual  operational  situations.  Well-planned  embedded  training  can  also  strengtnen  the  training 
process  in  other  ways,  such  as  by  providing  trainee  performance  assessment,  immediate 
feedback,  and  specific  remediation  geared  toward  the  immediate  job  environment.  On  the 
downside,  embedded  training  may  cause  additional  equipment  wear-and-tear  and  may  even 
pose  safety  or  security  hazards. 


Approach 


The  present  effort  had  two  goals.  The  Phase  I  goal  was  to  assess  the  potential  of 
embedded  training  for  the  future  Air  Force.  In  Phase  II,  the  goal  was  to  devise  a  viable 
embedded  training  R&D  strategy  for  AFHRL.  This  phase  represented  an  attempt  to  apply 
private  sector  strategic  planning  methods  to  a  military  laboratory.  The  focus  of  private  sector 
strategic  planning  methods  is  on  creating  and  sustaining  a  position  of  competitive  advantage 
in  the  marketplace.  Su^h  planning  usually  involves  considering  the  customers’  needs,  one’s 
own  sirengths,  and  the  strengths  of  competitois. 

Because  AFHRL  is  a  Government  research  agency,  not  all  private  sector  strategic  planning 
methods  are  applicable  to  it.  AFHF’L  is  not  a  private  sector  organization  which  competes  for 
business  in  the  open  marketplace.  However,  there  are  numerous  analogous  situations  where 
AFHRL  must  "compete"  for  its  "customers'"  business;  that  is,  it  must  perform  R&D  which  other 
Air  Force  agencies  or  major  commands  are  interested  in  and  are  v  'Ming  to  support.  Like  the 
business  plans  of  most  successful  businesses,  AFHRL’s  embedded  training  R&D  strategy  must 
take  into  account:  (a)  the  customers’  needs,  whether  perceived  by  the  customer  or  not;  (b) 
other  organizations  within  the  Government  or  in  the  private  sector  who  can  provide  embedded 
training  R&D  (potential  competitors  or  teammates);  and,  (c)  AFHRL’s  relevant  R&D  capabilities. 


Phase  I 


During  the  first  phase  of  the  study,  the  gotl  was  to  assess  trends  in  embedded  training 
throughout  the  military  services.  To  accomplish  this  end,  we  developed  a  database  of  embedded 
training  systems  already  in  use  or  being  developed  by  the  military.  We  contacted  selected 
individuals  involved  in  embedded  training  at  research  organizations,  acquisition  agencies,  arl 
system  development  organizations.  The  information  obtained  was  used  to  establish  the  database 
and  to  verify  the  accuracy  of  its  contents.  A  literature  review  was  conducted  from  which  we 
identified  approximate1.  68  different  Air  Force,  Army,  and  Navy  systems.  The  embedded  training 
database  includes  information  about  each  system’s  sponsor,  platform,  application,  development 
status,  configuration,  and  features.  (Appendix  A  shows  the  kind  of  information  contained  in 
the  database.) 

Phase  I  also  aimed  to  determine  what  considerations  impacted  decisions  to  use  or  not 
use  embedded  training.  This  issue  was  addressed  by  studying  further  14  of  the  more  promising 
systems.  A  set  of  selection  criteria  was  established  which  took  into  account  those  items  of 
interest  to  AFHRL.  Most  of  the  systems  selected  were  Air  Force  systems  in  various  stages 
of  development  or  fielding.  Therefore,  the  criteria  considered  that  some  of  the  systems  should 
have  entered  engineering  development  within  the  last  7  years;  a  few  of  the  systems  must  be 
ones  to  which  an  embedded  training  capacity  had  L>een  added  after  they  entered  operational 
use;  most  of  the  systems  should  bo  Air  Force  or  joint  projects  with  some  other  service;  and, 
a  cross-section  of  systems  should  be  included  (eg.,  communications,  radar,  management 


3 


information  systems,  command  and  control).  The  systems  listed  in  Table  1  were  selected  for 
further  study. 


Table  1.  Phase  I  Systems  Selected  for  Further  Study* 


Air  Force 

Army 

Navy 

AWACS 

Patriot 

ACTS 

ASAS/ENSCE 

ATF 

COOF 

F-16  ET 

F-16  OBEWS 

OBS  F-15 

PACCITS 

TEMPLAR 

WWMCSS 

TIU 

SETS 

*The  database  contains  detailej  descriptions  of  each  of  these  systems. 


The  authors  interviewed  the  personnel  responsible  for  each  system  to  find  out  what 
considerations  impacted  their  decisions  to  use  or  not  use  embedded  training.  A  questionnaire 
was  developed  and  used  to  provide  some  structure  to  the  interviews  (see  Appendix  B).  Typically, 
a  contact  involved  an  initial  phone  call  to  gather  preliminary  information  and  establish  a  basis 
for  further  discussion  with  either  the  initial  contact  or  others  involved  in  the  acquisition  or 
development  of  the  system.  During  interviews,  a  copy  of  the  completed  questionnaire  was 
normally  provided  to  the  interviewee  to  check  for  accuracy  and  to  further  direct  their  attention 
to  the  information  we  felt  would  be  of  interest  to  AFHRL.  Respondents  were  always  allowed 
to  deviate  from  the  questionnaire  it  they  had  something  to  say  about  embedded  training.  Often 
these  deviations  resulted  in  fascinating  anecdotal  information  about  the  system  which,  when 
coupled  witn  data  from  other  sources  and  other  systems,  helped  to  form  our  total  impression 
of  Air  Force  embedded  training.  Because  of  the  small  sample  involved  and  the  qualitative 
nature  of  the  data,  formal  statistical  analysis  techniques  were  not  used.  Chapter  II  contains 
the  results  of  Phase  I. 


Phase  II 


The  focus  of  the  second  phase  of  the  study  was  on  developing  a  strategic  plan  for 
embedded  training  research  at  AFHRL.  Developing  a  viable  strategy  requires  understanding 
potential  users'  needs,  assessing  the  competition,  and  building  on  the  organization’s  relative 
strengths.  The  purpose  is  to  build  on  an  organization’s  strengths  to  meet  its  customers’  and 
potential  customers'  needs.  Therefore,  in  the  second  phase  of  the  study,  we  identified  potential 
embedded  training  R&D  customers  for  AFHRL.  We  conducted  site  visits  to  selected  Air  Force 
organizations  to  determine  what  kind  of  problems  Air  Force  users  are  having  with  embedded 
training.  The  problems  were  examined  in  light  of  AFHRL  and  its  potential  competitors’  (i.e., 
private  sector  companies  and  other  DoD  organizations)  relative  strengths  at  solving  them.  After 
analyzing  this  information,  we  developed  an  embedded  training  R&D  strategy  for  AFHRL. 

As  a  starting  point  we  used  the  contacts  established  during  Phase  I  to  identify  potential 
embedded  training  customers.  Air  Force  organizations  judged  to  have  a  high  likelihood  of 
involvement  in  embedded  training  were  initially  contacted  by  phone.  Visits  were  made  to  those 
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organizations  which  seemed  to  be  the  most  promising  candidates  as  potential  customers. 
During  the  site  visits,  personnel  from  the  following  Air  Force  organizations  were  interviewed: 

Aeronautical  Systems  Division: 

Support  Systems  Engineering,  Training  Systems  Division  (ASD/ENET) 

Training  Systems  SPO,  Directorate  of  Tactical  &  Trainer  Training  (ASD/YWF) 

C-17  SPO,  Directorate  of  Test,  Deployment,  and  Training  Systems  (ASD/YCT) 

Electronic  Systems  Division: 

Intelligence  and  C3CM  Systems,  Automated  Message  Handling  System  (ESD/ICL) 

Tactical  Systems,  Over-the-Horizon  Backscatter  Radar  (OTH-B)  (ESD/TCO) 

Tactical  Systems,  Modular  Control  Equipment  (MCE)  (ESD/TCR) 

Joint  STARS  Program  Office  (ESD/JS) 

Space  and  Missile  Warning,  PAVE  PAWS  (ESD/SRP) 

U.S.  E-3  Program  Office  (ESD/YWP) 

Air  Force  Space  Command: 

Space  Operations  Training  and  Stan/Eval  (AFSPACECOM/DOT) 

Training  Plans  Division  (AFSPACECOM/XPTT) 

NORAD  Training  (NORAD/J3T) 

1013th  Combat  Crew  Training  Squadron: 

Satellite  Operations  Training  (1013  CCTS/DO) 

NORAD  Command  Post  Crew  IQT  (1013  CCTS/JO) 

Training  Equipment  Maintenance  (1013  CCTS/LK) 

Air  Force  Standard  Systems  Center: 

Cargo  Movement  Operation  System  Division  (SSC/AQFT) 

CAMS  Program  Management  Office  (SSC/AQMD) 

Medical  Systems  Program  Office  (SSC/AQS) 

Combat  Ammunition  System  Program  Office  (SSC/AQWS) 

Directorate  of  Comptroller  Systems  (SSC/SMC) 
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Directorate  of  Engineering  and  Transportation  Systems  (SSC/SML) 

Technical  Support  Division  (SSC/SSBT) 

Professional  Development  Division  (SSC/KST) 

ATC  Training  Advisory  Office  (SSC/TTA) 

Plans  and  Requirements  Office  (SSC/XP) 

A  questionnaire  was  developed  to  guide  these  interviews  (see  Appendix  C).  Our  assessment 
of  customer  embedded  training  R&D  needs  will  be  presented  in  Chapter  III. 

Our  assessment  of  AFHRL's  relative  strengths  with  regard  to  embedded  training  R&D  is 
based  on  a  review  of  the  Laboratory's  projects.  We  surveyed  the  completed  or  in-process 
projects  of  each  AFHRL  division  and  matched  the  capabilities  demonstrated  with  those  needed 
to  perform  the  requisite  embedded  training  R&D.  Our  assessment  of  AFHRL’s  competitors’ 
relative  strengths  is  based  on  information  and  opinions  gathered  during  the  course  of  this 
study  from  various  Government  representatives  involved  in  the  acquisition  and  development  of 
embedded  training  systems.  We  also  interviewed  several  industry  representatives  from  the 
private  sector  companies  represented  at  the  11th  Interservice/Industry  Training  Systems 
Conference.  There  were  five  presentations  on  embedded  training  at  the  conference.  We  were 
able  to  discuss  specific  embedded  training  systems  with  the  presenters.  Our  assessment  of 
AFHRL’s  and  its  competitors’  relative  strengths  appears  in  Chapter  IV. 

Our  overall  goal  in  formulating  an  R&D  strategy  for  AFHRL  was  to  identify  a  competitive 
"niche"  for  the  Laboratory  within  the  wide  range  of  R&D  which  can  be  performed.  Another 
way  of  looking  at  our  approach  is  shown  in  Figure  1.  That  figure  shows  that  some  Air  Force 
"customer  needs”  can  be  met  by  R&D.  Of  the  "researchable"  customer  needs,  those  that  are 
matched  by  AFHRL’s  relative  strengths  in  R&D  (shaded  area  in  Figure  1)  will  be  targeted  by 
the  R&D  strategy  which  we  have  formulated  for  AFHRL.  The  details  of  the  proposed  AFHRL 
R&D  strategy  are  discussed  in  Chapter  V. 


Figure  1.  AFHRL  R&D  Strategy. 
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II.  FUTURE  USES  OF  EMBEDDED  TRAINING  IN  THE  AIR  FORCE 


Phase  I  Findings 


During  Phase  I  of  the  study,  we  analyzed  the  state  of  embedded  training  in  the  Air  Force 
and  made  projections  regarding  future  uses  of  embedded  training.  This  chapter  discusses  the 
various  applications  of  embedded  training  in  four  major  areas  of  interest  to  the  Air  Force: 
embedded  training  in  signal  processing,  aboard  aircraft,  for  maintenance,  and  in  general  purpose 
computer  systems.  It  also  discusses  how  embedded  training  is  being  used  to  train  people  in 
the  Air  Force,  and  why  it  is  being  used  the  way  it  is. 


Applications  for  Embedded  Training 


Currently,  some  form  of  embedded  training  (simulation,  tutorial,  or  job  aid)  appears  in  a 
wide  variety  of  computerized  military  systems.  The  precise  form  of  the  embedded  training 
tends  to  vary  considerably  with  the  specific  computerized  application.  For  example,  simulation 
embedded  training  tends  to  be  used  within  strategic  systems  such  as  PAVE  PAWS  and  the 
NORAD  Command  Post  systems.  Even  some  current  tactical  systems  which  started  out  as 
strategic  systems  have  simulation  embedded  training  capabilities  (e.g.,  AWACS).  General 
purpose  computer  support  systems  such  as  the  Medical  Personnel  Records  System  (MPRS) 
and  the  Base  Contracting  Automated  System  (BCAS)  tend  to  make  use  of  either  job  aid  or 
tutorial  embedded  training,  or  both. 

The  use  of  job  aid  or  tutorial  embedded  training  for  support  applications  is  not  surprising 
in  that  most  support  applications  run  on  general  purpose  computers  which  can  accommodate 
a  wide  variety  of  software  from  training  to  databases,  etc.  However,  it  is  far  more  interesting 
to  note  when  and  why  embedded  training  is  included  in  the  more  specialized  systems,  due 
to  the  fact  that  presumably  it  is  more  difficult  to  add  a  training  capability  to  these  systems. 


Embedded  Training  in  Signal  Processing 


We  found  that  within  specialized  computer  systems,  embedded  training  is  most  likely  to  be 
found  in  those  systems  that  involve  some  form  of  signal  processing.  Signal  processors  are 
systems  which  require  a  human  operator  to  interpret  and  respond  to  some  type  of  video  or 
audio  signal,  such  as  radar  or  sonar  systems.  They  typically  involve  reading  signals  from  a 
scope  and  properly  classifying  these  signals  as  threats  or  not.  Of  the  68  specialized  computer 
systems  identified  in  the  first  phase  of  this  study,  58  (almost  90%)  fell  into  this  category  (see 
Figure  2).  Many  of  the  Navy  systems  studied  can  be  classified  as  signal  processing  systems. 
There  was,  however,  a  significant  difference  between  the  Air  Force  and  the  Navy  when  it  came 
to  the  system  operators.  Navy  shipboard-type  sensor  systems  (usually  found  in  a  ship's  combat 
control  center)  are  normally  tactical  systems  which  are  routinely  operated  by  enlisted  technicians. 
On  the  other  hand,  similar  Air  Force  systems  can  be  either  tactical  (e.g,,  C2)  or  strategic  (e.g., 
space  warning  and  sensor)  systems.  The  Air  Force  systems  can  be  operated  by  officers  or 
enlisted  technicians.  The  aircraft  systems  studied  (AWACS  and  F-16  OBEWS)  are  more 
frequently  operated  by  officers.  In  both  the  Air  Force  and  Navy  cases,  the  embedded  training 
capability  allows  the  entire  operational  team  to  practice  making  those  decisions  they  would  be 
required  to  make  under  wartime  or  threat  conditions.  The  other  computerized  systems 
investigated  were  evenly  distributed  across  several  other  categories  such  as  intelligence, 
communications,  and  command  and  control  decision  aids,  etc. 
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HSP 

Human  Signal  Processing 

Supp  -- 

Financial  Support 

C&C 

Command  and  Control 

Intel 

Intelligence  Gathering 

Comm 

Communications 

Misc 

Miscellaneous 

We  believe  that  embedded  training  capabilities  in  the  Air  Force  will  continue  to  be  found 
most  often  in  signal  processing  systems  in  the  future.  There  appear  to  be  two  reasons  for 
this  trend.  First,  the  technology  necessary  to  embed  a  training  capability  into  an  operational 
system  is  often  already  available  as  part  of  these  systems.  Contractors  routinely  build-in 
system  test  functions  to  help  evaluate  these  systems  prior  to  Government  acceptance.  These 
functions  can  often  be  used  or  modified  for  training.  Indeed,  respondents  from  those  programs 
with  an  embedded  training  capability  frequently  referred  to  the  capability  as  the  "system  test" 
function  rather  than  as  the  embedded  training  function.  In  some  cases,  only  after  the  system 
was  fielded  was  it  recoonized  that  this  system  test  function  provided  an  excellent  opportunity 
to  train  new  operators  and  maintain  the  proficiency  level  of  experienced  operators. 

A  second  reason  for  the  trend  to  include  an  embedded  training  capability  in  signal  processing 
systems  is  that  the  ability  to  recognize  and  respond  properly  to  a  threat  is  a  highly  critical 
skill  which  requires  frequent  testing  and  practice  under  emergency  conditions.  Pilots  and  radar, 
sonar,  and  electronic  warfare  operators  need  to  be  able  to  respond  immediately  and  correctly 
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in  threat  situations.  Embedded  training  allows  operators  of  these  systems  to  practice  wartime 
procedures  under  simulated  wartime  situations;  therefore,  there  may  be  a  greater  impetus  to 
embed  a  training  capability  into  these  systems  than  in  other  types  of  specialized  computer 
systems. 


Embedded  Training  Aboard  Aircraft 


Embedded  training  is  found  much  less  frequently  aboard  aircraft  than  it  is  found  in  shipboard 
or  ground-based  systems.  However,  several  Air  Force  personnel  at  Tactical  Air  Command 
(TAC)  and  the  Aeronautical  Systems  Division  (ASD)  have  expressed  an  interest  in  embedding 
training  within  aircraft  systems.  After  all,  aircraft  do  utilize  signal  processing  systems  with 
video  and  audio  signals.  Yet,  only  two  airborne  embedded  training  systems  have  been  fielded: 
the  F-16  OBEWS  and  AWACS.  Three  other  aircraft  embedded  training  systems  (for  the  F-15, 
the  F-16,  and  the  Advanced  Tactical  Fighter)  are  currently  in  the  planning  stages;  however, 
they  have  met  considerable  resistance  from  some  quarters  within  TAC  and  ASD. 

The  resistance  is  due  to  concerns  about  limited  on-board  computer  memory  capacity  and 
weight  limitations.  Weight  limitations  could  constrain  the  addition  of  a  pod  like  the  F-16 
OBEWS.  Many  aircraft  acquisition  personnel  want  to  use  every  additional  ounce  to  maximize 
the  performance  of  the  aircraft.  They  perceive  embedded  training  as  an  unneeded  extra, 
because  other  acceptable  ways  to  address  the  training  requirements  are  available.  One  Air 
Force  acquisition  manager  has  said  that  embedded  training  "weighs  too  much!"  Whether  true 
or  not,  concerns  about  added  weight  kept  embedded  training  off  aircraft. 

The  AWACS  and  F-16  OBEWS  are  not  really  exceptions  to  this  trend.  The  AWACS 
embedded  training  component  is  not  an  aircraft  system  per  se;  rather,  it  is  part  of  a  raaar 
system  which  happens  to  be  aboard  an  aircraft.  Similarly,  the  F-16  OBEWS  is  a  pod  which 
is  attached  to  an  F-16  tor  Electronic  Warfare  training.  It  can  be  removed  quickly  when  the 
aircraft  needs  to  be  combat  ready. 


Embedded  Training  for  Maintenance 


Embedded  training  has  not  been  used  for  maintenance  training.  After  much  searching,  we 
were  able  to  make  contact  with  only  one  individual  who  could  identify  a  maintenance  embedded 
training  system-the  Automated  Ground  Engine  Test  System  (AGETS).  The  embedded  training 
component  for  this  system  turned  out  to  be  a  "page-turner"  tutorial  which  introduces  the  test 
system  to  new  users.  It  does  not  provide  training  for  maintenance  technicians  in  troubleshooting 
problems  with  the  system  or  making  repairs;  that  is,  it  is  not  used  for  training  the  performance 
of  maintenance  tasks.  Based  on  our  inability  to  find  any  evidence  from  the  literature  search 
or  through  personal  interviews  of  embedded  maintenance  training  systems,  it  appears  at  this 

time  that  embedded  training  is  not  being  used  to  teach  maintenance  skills.  This  does  not 

imply  that  embedded  training  is  incapable  of  being  used  for  training  maintenance  skills,  but 

that  Air  Force  training  managers  and  computer  system  designers  and  developers  appear  to 
have  ruled  it  out  for  other  reasons. 

There  are  three  possible  explanations  for  the  lack  of  embedded  training  systems  for 

maintenance  training.  First,  we  found  a  widespread  concern  among  the  people  we  contacted 
about  accidental  insertion  of  faults  into  operational  equipment  while  using  that  equipment. 
Second,  all  the  Air  Training  Command  (ATC)  training  managers  whom  we  interviewed  believed 
there  was  some  policy  that  prohibited  embedded  maintenance  training.  However,  we  could 
not  obtain  a  copy  of  the  policy.  A  third  reason  might  be  that  maintenance  skills  do  not  lend 
themselves  to  being  taught  through  embedded  training  as  easily  as  other  skills;  training  such 
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skills  as  troubleshooting  might  require  a  different  approach.  It  may  well  be  that  embedded 
training  works  best  for  procedural  skills.  This  is  yet  to  be  determined  by  research.  We  are 
forced  to  conclude  that,  if  present  trends  continue,  embedded  training  is  not  likely  to  be  used 
to  teach  maintenance  skills  in  the  future. 


Embedded  Training  in  General  Purpose  Computer  Systems 


As  mentioned  previously,  embedded  training  in  general  purpose  computer  systems  takes 
the  form  of  tutorial  or  job  aiding  rather  than  simulation.  This  is  probably  due  to  the  fact  that 
general  purpose  applications  do  not  require  rehearsal,  as  signal  processing  applications  do. 
There  is  not  the  criticality  for  an  immediate  response  to  a  signal  or  threat  that  there  is  with 
"operational"  systems.  Although  there  may  be  a  real  need  for  better  quality  training  for  general 
purpose  computer  systems,  it  usually  takes  the  form  of  a  tutorial  or  job  aid.  These  are 
considerably  less  expensive  to  develop  than  job  simulations. 

We  found  that  embedded  training  was  frequently  planned  for  a  general  purpose  system  but 
not  developed.  For  example,  the  Comptroller’s  Office  of  the  Future  (COOF)  program  originally 
planned  to  add  an  embedded  training  component  to  its  accounting  subsystem.  However,  during 
the  software  development  process,  the  embedded  training  component  was  left  out.  As  a  result, 
instruction  on  the  system  is  currently  being  carried  out  by  Mobile  Training  Teams  (MTTs)  from 
ATC.  Our  interviews  with  an  Accounting  and  Finance  Center  System  2200  program  manager 
indicated  that  the  embedded  training  component  was  deleted  because  of  a  lack  of  resources. 
The  System  2200  software  development  organization  also  felt  that  the  addition  of  embedded 
training  would  have  degraded  system  performance. 

In  summary,  we  found  that  embedded  training  can  be  used  within  both  specialized  and 
general  purpose  computer  systems.  Within  specialized  systems,  embedded  training  tends  to 
be  used  most  frequently  within  signal  processing  systems  such  as  radar  or  sonar  systems. 
Aircraft  systems  are  an  exception  to  this  rule.  This  is  probably  attributable  to  concerns  about 
embedded  training  consuming  the  aircraft’s  limited  resources.  In  general,  it  is  much  easier  to 
add  embedded  training  capabilities  to  general  )urpose  computer  systems.  However,  the  impetus 
to  do  so  is  not  as  great  as  with  specialized  systems.  Finally,  whatever  the  reason,  embedded 
training  tends  not  to  be  used  for  maintenance  training  at  this  time.  We  expect  these  trends 
in  embedded  training  applications  to  continue  in  the  future. 


Use  of  Embedded  Training 


Because  the  embedded  training  capability  is  by  definition  built  into  operational  equipment, 
it  tends  to  be  used  for  on-the-job  training.  The  users  of  embedded  training  are  usually  qualified 
operators.  They  do  not  have  a  requirement  for  training  new  skills.  The  primary  use  of  most 
embedded  training  systems  is  to  maintain  operational  readiness  proficiency.  It  keeps  operators 
ready  to  do  their  jobs. 

Many  of  the  personnel  interviewed  for  this  study  had  never  heard  the  term  "embedded 
training"  applied  to  the  system  capability  they  were  using  for  proficiency  training.  They  referred 
to  it  as  an  "exercise  capability"  or  "simulation  capability."  In  their  minds,  the  term  "training” 
applies  to  the  schoolhouse  or  to  initial  qualification  training. 

The  effect  of  an  embedded  training  component  can  extend  beyond  the  actual  computerized 
system  operator.  For  many  signal  processing  systems,  the  embedded  training  capability  is 
used  not  only  by  the  operators  sitting  at  the  consoles,  but  also  by  their  fellow  team  members 
involved  in  using  the  information  and  making  decisions  based  on  that  information.  In  our 
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interviews,  we  found  that  although  embedded  training  can  be  used  for  individual  operator 
training,  it  is  most  frequently  used  for  team  training  exercises.  For  example,  NORAD  Command 
Post  exercises  are  used  not  only  to  train  technicians  to  operate  their  consoles,  but  also  to 
train  commanders  in  the  decision-making  and  coordination  procedures  that  they  would  need  to 
follow  during  actual  operations.  With  the  appropriate  communications  hookups,  even  inter-site 
or  multi-platform  training  is  possible.  The  Navy  is  currently  making  use  of  such  communications 
links  to  extend  training  to  multiple  ships  participating  in  limited  combat  control  center  exercises. 

Finally,  we  noted  that  embedded  training  does  not  seem  to  fall  under  the  rubric  of  formal 
Air  Force  training.  We  were  unable  to  find  any  Air  Force  policy  which  addressed  the  way(s) 
in  which  embedded  training  should  be  used  either  on-the-job  or  in  formal  school  training  or 
how  it  is  to  fit  into  a  total  training  picture.  Air  Training  Command  does  not  have  such  a 
policy  at  the  headquarters  level,  nor  could  we  find  any  policy  guidance  at  the  Technical  Training 
Centers.  Air  Training  Command  seems  to  consider  embedded  training  a  system-specific 
capability  for  which  the  Major  Commands  (MAJCOMs)  have  responsibility. 

In  summary,  we  found  that  embedded  training  is  commonly  regarded  by  its  users  as  a 
system  capability  useful  for  operational  readiness  training  of  experienced  operators  and  teams. 
There  does  not  appear  to  be  any  Air  Force  policy  governing  its  use,  employment,  or  development 
as  a  training  medium.  Decisions  about  its  use  tend  to  be  made  in  the  field,  often  at  the  unit 
level. 


Development  of  Embedded  Training 


Because  embedded  training  tends  to  be  regarded  more  as  a  system  capability  than  as  a 
training  medium,  it  is  developed  in  a  different  way  than  are  dedicated  training  systems.  Indeed, 
the  embedded  training  components  of  most  systems  are  not  ‘‘developed"  in  the  sense  that 
trainers  use  the  term.  Training  system  development  normally  involves  using  some  form  of  the 
Instructional  Systems  Development  (ISD)  process.  This  frequently  begins  with  the  development 
of  a  job  and  task  analysis,  from  which  training  requirements  are  derived  and  specifications  for 
the  system  are  established.  Training  objectives  are  developed  along  with  evaluation  standards 
and  test  items.  Finally,  the  appropriate  media  (which  would  presumably  include  embedded 
training  as  an  alternative)  are  recommended  and  selected  based  on  cost  and  trade-off  analyses. 
In  contrast,  embedded  training  systems  development  may  range  from  simple  enhancement  of 
the  existing  system’s  test  capability  (e.g.,  PAVE  PAWS  and  BMEWS)  to  full-blown  use  of  the 

ISD  process  (e.g.,  ACTS,  SETS).  However,  the  former  approach  is  more  typical. 

The  developers  of  embedded  training  systems  tend  to  have  different  backgrounds  from 
those  of  other  training  system  developers.  Contractors  which  are  presently  developing  systems 
with  embedded  training  are  often  not  training  systems  or  services-oriented  companies.  They 
tend  to  be  developers  of  electronics  and  communications  systems,  such  as  Boeing,  Litton,  and 
Raytheon.  For  such  contractors,  developing  embedded  training  is  more  of  an  engineering  task 
than  a  training  development  task.  Within  the  Air  Force,  personnel  from  the  user  (operational) 
commands  are  frequently  involved  in  the  discussions  leading  up  to  the  design  of  a  system’s 

embedded  training  capability.  Air  Force  training  specialists  are  rarely  involved  in  the  design 

of  an  embedded  training  capability. 

Finally,  unlike  dedicated  training  systems,  embedded  training  is  not  costed  separately  from 
its  host  system.  Many  of  the  personnel  we  interviewed  told  us  that  they  could  not  give  a 
figure  for  the  cost  of  the  embedded  training  component(s)  of  a  system.  Presumably,  because 
the  embedded  training  capability  was  considered  part  of  the  host  system,  there  were  no  cost 
figures  available.  In  a  large-scale  procurement,  the  embedded  training  capability  is  considered 
an  extension  of  the  system  test  capability. 
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In  summary,  the  development  of  embedded  training  is  handled  very  differently  from  the 
development  of  dedicated  training  systems.  In  some  cases,  the  inclusion  of  embedded  training 
in  a  system  is  purely  serendipitous.  When  it  is  purposefully  included  in  a  computerized  system, 
it  is  viewed  more  as  a  means  for  practice  of  essential  skills  than  as  a  means  for  acquiring 
new  skills.  We  have  every  reason  to  believe  that  this  trend  will  continue  in  the  future. 


Embedded  Training  System  Architectures 


There  are  two  distinct  embedded  training  architectures.  Embedded  training  systems  may 
be  either  built  into  a  host  system  ("fully  embedded")  or  attached  to  the  host  via  a  separate 
piece  of  hardware  ("strapped  on").  Each  type  of  embedded  training  has  its  advantages  and 
disadvantages.  For  example,  strap-on  embedded  training  components  are  normally  added  to 
existing  systems.  This  can  be  disadvantageous  in  some  instances,  particularly  in  the  case  of 
aircraft  embedded  training  systems.  Some  specific  objections  to  strap-on  embedded  training 

systems  for  aircraft  have  been  voiced  by  Gershanoff  (1989).  He  reports,  "One  argument  voiced 
against  the  OBEWS  pod  approach  is  that  it  would  not  be  as  good  as  an  embedded  training 
system,  where  the  circuits  that  geneiate  threat  signals  and  training  scenarios  are  built  into  the 
aircraft  and  perhaps  the  operational  avionics  and  EW  gear  themselves."  A  more  realistic 

reason  for  pilots’  objections  to  OBEWS  and  pods  in  general  may  be  that  .  .  pods  of  any 
kind  are  not  the  favorite  things  of  pilots,  who  claim  that  they  can  cause  asymmetric  loading 
when  mounted  on  one  of  the  nine  F-16  weapon  stations,  and  that  they  may  impair  aircraft 
handling"  (Gershanoff,  1989).  Other  objections  to  strap-ons  may  be  that  they  add  weight  to 
a  system,  they  may  cause  additional  equipment  wear-and-tear  because  they  are  repeatedly 
connected  and  disconnected,  and  they  are  themselves  another  piece  of  equipment  to  acquire 
and  maintain.  However,  fully  built-in  embedded  training  is  sometimes  not  the  best  or  most 

convenient  solution  to  a  problem.  For  example,  given  the  space  and  computer  memory 

limitations  in  small  tactical  aircraft,  strap-on  systems  like  the  F-16  OBEWS  represent  the  most 
viable  way  of  embedding  training  for  the  pilots  of  these  aircraft. 

Normally,  one  might  expect  a  built-in  to  be  better  than  a  strap-on  embedded  training  system. 
However,  built-in  systems  can  have  disadvantages  of  their  own.  For  example,  our  interviews 
showed  that  built-in  systems  often  fail  to  capitalize  on  previous  work  in  selecting  the  needed 
embedded  training  features  (i.e.,  playback,  performance  recording,  etc.).  One  built-in  system 
is  frequently  as  different  from  another  built-in  system  as  from  a  strap-on  system. 

In  summary,  we  believe  that  both  strap-on  and  built-in  embedded  training  systems  will 
continue  to  be  used.  Strap-on  components  will  continue  to  be  used  for  embedding  training 
in  existing  systems.  Built-in  embedded  training  will  probably  be  used  for  developing  systems. 
We  believe  that  strap-on  embedded  training  systems  probably  represent  the  wave  of  the  future 
for  aircraft ,  because  they  do  not  consume  aircraft  space  or  resources.  On  the  other  hand, 
large  computerized  ground-based  systems  will  probably  tend  to  be  fully  embedded  (i.e.,  built-in). 


Future  Improvements  to  Embedded  Training 


There  are  several  technological  developments  which  could  impact  embedded  training  in  the 
future.  Improved  speed  and  memory  capacity,  artificial  intelligence  programming  techniques, 
and  enhanced  graphics  displays  could  all  enhance  training  (Massey,  Harris,  Downes-Martin,  & 
Kurland,  1986). 

Few  institutional  forces  are  driving  the  development  of  embedded  training  technology.  It 
is  not  clear  that  users  of  embedded  training  are  looking  for  either  improved  embedded  training 
or  improved  technology  for  embedded  training.  As  mentioned  earlier,  many  Air  Force  users 
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of  embedded  training  are  not  aware  that  they  are  using  embedded  training.  They  view  the 
embedded  training  function  they  have  not  as  training,  but  as  a  system  capability  which  provides 
them  with  an  inexpensive  and  convenient  means  of  maintaining  skill  proficiency.  Moreover, 
users  are  not  particularly  dissatisfied  with  the  capabilities  they  have.  They  really  don’t  know 
rf  anything  better  is  available. 


ill.  POTENTIAL  CUSTOMERS’  NEEDS 


Site  Visit  Findings 


Most  of  our  information  regarding  problems  which  are  being  encountered  with  embedded 
training  has  come  from  data  gathered  during  our  visits  to  four  separate  Air  Force  organizations. 
This  section  summarizes  what  we  learned  about  AFHRL's  potential  customers  and  the  customers’ 
perceived  needs  regarding  embedded  training.  First,  we  will  summarize  our  findings  by 
organization;  then  we  will  group  our  fine  ngs  into  significant  problem  areas.  The  problems 
described  in  this  section  may  not  all  have  research  solutions.  However,  our  intent  is  to  first 
present  a  full  menu  of  problems,  and  then  to  identify  those  that  AFHRL  can  potentially  address. 


Aeronautical  Systems  Division 


At  the  start  of  our  visit  to  ASD,  we  were  told  that  embedded  training  had  been  proposed 
for  the  F-15,  the  F-16,  the  C-17  and  the  Advanced  Tactical  Fighter  (ATF)  aircraft.  However, 
in  subsequent  discussions  with  the  C-17  System  Program  Office  (SPO),  its  staff  said  that  they 
knew  nothing  about  an  embedded  training  capability. 

Little  embedded  training  information  has  been  released  regarding  the  ATF.  We  suspect 
that  little  might  have  been  done  to  embed  training  capabilities  into  its  operational  systems. 
The  specifications  for  the  ATF  require  only  that  embedded  training  be  investigated;  they  do 
not  stipulate  that  specific  training  requirements  be  addressed  by  embedded  training. 

Of  the  two  active  aircraft  (F-15  and  F-16),  only  the  F-16  has  a  fielded  embedded  training 

system -OBEWS.  However,  OBEWS  is  being  developed  by  the  Munitions  Systems  Division 

(MSD),  rather  than  by  ASD.  The  OBS  F-15  (On-Board  Simulator)  is  currently  in  the  experimental 
stages;  it  consists  of  an  on-board  simulation  capability  embedded  in  an  experimental  Integrated 
Flight/Fire  Control  (IFFC)  flight  test  program  for  the  F-15.  The  ASD  Plans  and  Programs 
Office’s  Embedded  Training  for  Tactical  Aircraft  project  is  also  in  the  early  concept  investigation 
stages.  This  ET  system,  which  is  intended  to  train  EW  skills,  at  this  time  consists  of  a  ground 
simulator  which  is  being  used  for  pilot  testing;  that  is,  it  has  not  yet  flown. 

During  our  interviews  we  found  definite  interest  in  embedded  training  research  in  some 

quarters  at  ASD.  The  Plans  and  Programs  Office  (ASD/XR)  is  presently  investigating  the 
feasibility  of  embedding  training  aboard  tactical  aircraft;  however,  ASD/XR  does  not  appear  to 
be  a  potential  AFHRL  customer.  Individuals  from  the  Wright  Research  and  Development  Center, 
Flight  Dynamics  Laboratory  (WRDC/FIGX)  and  the  ASD  Engineering  Division  (ASD/ENET)  have 
already  approached  AFHRL  to  request  research  assistance  on  embedded  training  using  the 
F-15  OBS. 

Not  everyone  at  ASD  is  enthusiastic  about  embedded  training.  The  problem  seems  to  be 
that  many  aircraft  SPO  program  management  personnel  perceive  embedded  training  to  be  an 
added  burden  on  the  aircraft,  and  thus  are  reluctant  to  support  it.  There  is  a  concern  that 
embedded  training  will  add  weight  to  the  aircraft  or  will  consume  scarce  computer  resources. 
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According  to  one  interviewee  at  ASD/ENET,  the  following  considerations  are  key  issues  in 
embedded  training: 

1.  The  impact  of  embedded  training  on  avionics  complexity. 

2.  The  impact  of  embedded  training  on  avionics  reliability  and  maintainability. 

3.  Prevention  of  false  alarms  in  non-training  modes. 

4.  Potential  fixation  on  synthetic  targets  (i.e. ,  whether  the  synthetic  targets  generated  by 
the  embedded  training  system  will  cause  confusion  or  otherwise  distract  attention  from 
real  targets). 

5.  Acceptable  methods  for  implementing  safety  features. 

6.  Determination  of  the  requirements  of  the  end  users  of  embedded  training. 

7.  Types  of  real-time  feedback  needed. 

8.  Types  of  performance  measurement  needed. 

The  first  five  of  these  items  have  an  impact  on  system  safety.  ASD  appears  to  want 

answers  in  all  these  areas  before  it  can  begin  to  incorporate  embedded  training  in  the  systems 

it  is  acquiring  or  developing. 

According  to  ASD,  SPO  program  managers  need  assistance  in  evaluating  the  usefulness 
of  embedded  training  in  current  and  emerging  weapon  systems.  Currently  they  must  default 
to  traditional  OJT  or  use  CBT  as  a  substitute  because  they  cannot  properly  predict  whether 
embedded  training  is  an  acceptable  alternative.  A  legitimate  area  for  R&D  is  the  investigation 
of  embedded  training  as  an  acceptable  alternative  for  maintenance  training. 


Electronic  Systems  Division 


The  Electronic  Systems  Division  (ESD)  has  provided  an  embedded  training  capability  in  at 
least  some  systems  since  the  late  1950s.  However,  the  individuals  that  we  interviewed  from 
ESD  had  not  heard  the  term  "embedded  training"  used  to  describe  these  system  capabilities. 
Moreover,  the  ESD  personnel  we  talked  to  generally  did  not  perceive  themselves  to  be 
developers  of  "training."  Consequently,  although  ESD  might  benefit  from  embedded  training 
research,  the  division  must  be  approached  using  different  terms  (i.e.,  ones  that  they  can 
understand  and  relate  to). 

Not  all  program  offices  were  equally  cognizant  of  the  possibilities  of  embedding  training. 
Users  and  potential  users  of  embedded  training  at  ESD  could  benefit  from  knowing  how  other 
electronic  systems  have  successfully  used  embedded  training  and  how  the  capability  can  be 
built  into  an  operational  system.  For  example,  the  embedded  training  within  PAVE  PAWS  and 
BMEWS,  two  Space  and  Missile  Warning  Systems  developed  by  ESD/SR,  is  a  modified  version 
of  the  system  test  capability.  In  contrast,  the  program  managers  working  with  the  Over-the-Horizon 
Backscatter  (OTH-B)  Radar  System  told  us  that  adding  an  embedded  training  capability  to 
OTH-B  would  probably  be  too  expensive  because  it  was  technologically  too  difficult.  Yet, 
OTH-B  is  essentially  the  same  type  of  system  as  PAVE  PAWS  and  BMEWS.  Why  one  group 
of  program  managers  thought  embedded  training  was  technologically  infeasible  and/or  too 
expensive  while  the  other  group  included  the  capability  in  their  system  might  reflect  a  lack  of 
guidance  regarding  embedded  training.  The  OTH-B  program  managers  were  not  drawing  on 
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the  experience  of  the  BMEWS  or  PAVE  PAWS  program  managers.  The  point  is  that  there  is 
no  embedded  training  guidance  available  now  for  program  managers  to  help  them  make  iheir 
decisions.  Without  the  experience  of  similar  past  projects,  program  managers  have  no  basis 
for  making  informed  system  training  decisions. 


Air  Force  Standard  Systems  Center 


The  Air  Force  Communications  Command's  Standard  Systems  Center  (SSC)  is  responsible 
for  acquiring  computer  systems  which  will  be  used  by  two  or  more  MAJCOMs.  Systems 
acquired  by  SSC  generally  consist  of  some  combination  of  general  purpose  hardware  and 
broad  applications  software.  Many  of  SSC's  applications  packages  have  some  type  of  tutorial 
or  on-line  help  component  which  can  be  considered  embedded  training.  As  with  most  of  the 
other  Air  Force  organizations  we  visited,  none  of  the  SSC  personnel  we  spoke  to  had  heard 
the  term  "embedded  training"  used  to  describe  software  which  provides  training  in  the  form  of 
help  or  assistance. 

Because  the  SSC  acquires  general  purpose  rather  than  specialized  systems,  its  embedded 
training  needs  differ  dramatically  from  those  of  ESD  and  ASD.  As  we  noted  in  Chapter  II, 
the  embedded  training  in  specialized  computer  systems  is  usually  oriented  toward  teaching 
procedural  skills  to  operators  of  signal  processing  systems  and  is  normally  oriented  toward  the 
practice  of  emergency  or  wartime  skills.  The  embedded  training  capability  in  a  signal  processing 
system  is  used  to  prepare  the  operator  to  respond  quickly  and  correctly  in  threat  situations. 
For  operators  of  standard  systems,  the  need  to  act  quickly  and  correctly  is  not  critical.  Instead, 
users  of  standard  systems  most  often  need  to  be  trained  to  use  a  given  software  application. 

The  SSC’s  largest  training  problem  is  that  its  users  often  experience  difficulty  in  learning 
to  operate  the  systems  it  acquires.  At  least  two  interviewees  noted  that  for  whatever  reason, 
planning  for  training  tends  to  be  left  out  during  the  system  specification  development  process. 
It  was  reported  that  training  was  omitted  to  reduce  cost.  Also  development  managers  usually 
worked  with  the  initial  assumption  that  their  systems  would  be  so  user-friendly  that  training 
would  not  be  necessary.  Almost  inevitably,  when  a  new  system  arrived  in  the  field,  users  did 
not  know  what  to  do  with  it  or  how  to  operate  it.  Other  means  of  training  then  had  to  be 
provided.  They  ranged  from  adding  or  enhancing  on-line  helps  and  tutorials  to  providing 
traditional  instruction.  Most  users  had  a  preference  for  traditional  stand-up  instruction.  This 
was  probably  due  to  the  inability  of  most  user  manuals,  help  functions,  or  CBT  to  handle  the 
many  unique  individual  problems  and  questions  that  users  have. 

The  SSC  might  benefit  from  research  to  develop  intelligent,  context-sensitive  help  or  job 
aids  which  could  ease  users'  needs  for  instructors.  Currently  most  job  aids,  software  tutorials, 
or  helps  are  quite  insensitive  to  context  and  to  users'  perspectives.  For  example,  users  expect 
the  "help"  function  in  an  applications  program  to  provide  information  on  the  specific  portion  or 
function  of  the  program  being  used  at  the  time.  If  help  for  the  application  is  sandwiched 
between  several  layers  of  help  menus,  the  user  can  be  frustrated.  As  a  result,  traditional 
classroom  training  is  preferred.  Additional  research  might  be  performed  on  the  application  of 
intelligent  computer-assisted  instruction  (ICAI)  in  large  standard  systems.  Its  use  could  allow 
the  user  to  query  the  system  to  determine  precisely  what  is  needed  in  the  way  of  help  to 
perform  a  specific  function. 

The  SSC’s  most  immediate  need  appears  to  be  a  means  for  incorporating  informed  training 
planning  into  its  systems  acquisition  and  development  process.  This  planning  task  is  complicated 
by  the  fact  that  most  of  the  SSC’s  systems  are  used  by  two  or  more  major  commands.  The 
capabilities  and  resources  available  to  its  target  user  groups  may  vary  considerably.  As  an 
example,  the  Training  Planning  Team  (TPT)  for  the  Core  Automated  Maintenance  System 
(CAMS)  consists  of  representatives  from  the  MAJCOMs,  ATC,  and  SSC.  Although  this  group 


15 


makes  far-reaching  decisions  regarding  training  for  standard  systems  software,  they  must 

frequently  do  so  without  the  benefit  of  knowledge  of  research  on  what  kinds  of  training 

(especially  embedded  training)  are  best  suited  to  specific  application  problems. 

CAMS  also  provides  an  example  of  how  the  SSC  could  use  help  in  Total  Training  System 
planning.  For  example,  the  SSC  ATC  liaison  office  is  developing  CBT  training  for  CAMS  using 
Merlin  (a  CBT  authoring  system)  whereas  the  ATC  Technical  Training  Center  is  developing 
other  CBT  training  using  the  Saber  authoring  system.  Personnel  we  interviewed  at  the  SSC 

reported  that  there  were  at  least  five  separate  efforts  to  develop  CBT  for  CAMS. 

We  also  found  that  the  SSC  does  not  always  include  specific  training  requirements  in  the 

specifications  for  a  system.  The  SSC  had  determined  that  a  sophisticated  simulation  capability 

which  verged  on  embedded  training  was  necessary  for  training  on  the  Cargo  Movement  System 
(CMOS).  However,  this  requirement  was  not  included  in  the  system  specification  because  of 

the  perception  that  it  would  increase  system  costs.  Instead,  the  specification  included  a 

requirement  for  "orientation"  training  on  the  system.  As  with  the  OTH-B  program  at  ESD, 
better  information  about  the  costs  of  ET  could  h3ve  led  to  improved  training  decisions. 


AFSPACECOM 


AFSPACECOM  has  embedded  training  capabilities  in  almost  all  of  its  space  tracking  'md 
early  warning  systems.  The  use  of  embedded  training  is  a  very  important  part  of  its  aerospace 
operator  training  programs.  In  some  instances,  the  system's  embedded  training  component  is 
the  only  means  by  which  trainees  can  get  hands-on  practice  with  actual  operational  equipment. 

b.  me  AFSPACECOM  officials  are  wary  of  using  on-line  systems  for  training.  Currently, 
they  are  seeking  to  reduce  or  eliminate  the  use  of  embedded  training  from  the  AFSPACECOM 
training  program;  their  current  aim  is  to  use  full  and  part-task  simulators. 

One  of  the  reasons  for  AFSPACECOM  s  reluctance  to  use  embedding  training  in  operational 

systems  is  an  incident  in  which  simulated  signals  were  mistaken  for  real  ones.  As  a  result 

of  this  mishap,  simulation  capabilities  were  ordered  removed  from  all  space  surveillance  systems. 
Although  not  everyone  at  AFSPACECOM  believes  that  embedding  training  in  operational  systems 
is  necessarily  a  bad  idea,  there  is  a  definite  uneasiness  about  embedded  training.  Therefore, 
some  individuals  with’n  AFSPACECOM  would  not  be  very  receptive  to  embedded  training 
research 

However,  not  all  systems  are  prone  to  the  problem  of  simulated  signals  being  mistaken 
for  real  ones.  For  example,  AFSPACECOM  personnel  who  had  worked  with  the  IBM  4381 
system  at  Falcon  AFB  said  that  there  was  virtually  no  chance  of  simulated  signals  "contaminating" 
the  system.  If  the  segregation  of  simulated  data  from  real  data  can  be  achieved  in  this 
system,  it  could  be  achieved  in  others.  The  real  problem  may  lie  not  with  the  embedded 
training,  but  with  how  some  AFSPACECOM  systems  have  been  engineered  and  how  they  are 
used  for  training.  These  specific  AFSPACECOM  problems  need  research. 

A  secondary  source  of  reluctance  to  use  embedded  training  by  AFSPACECOM  reflects 
problems  with  scheduling  training  time  on  the  operational  equipment.  System  equipment  is 
often  in  high  demand,  not  only  for  operational  use  but  also  for  maintenance,  for  software 

development,  and  for  testing.  On-the-job  training  has  a  relatively  low  priority.  Frequently, 
trainees  must  deal  with  real-world  interruptions  during  training  exercises.  Some  managers 

within  AFSPACECOM  take  the  positions  that  off-line  simulators  would  make  more  training  time 
available  and  that  such  training  time  could  be  concentrated  on  specific  requirements. 
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Although  some  AFSPACECOM  managers  seem  opposed  to  continued  use  of  embedded 
training,  it  appears  that  the  Space  Systems  Division  (SSD)  has  taken  some  initiative  in  the 
area  of  developing  standards  for  the  various  types  of  simulation  ET.  One  AFSPACECOM 
interviewee  mentioned  that  SSD/CWOL  has  issued  a  draft  document  which  addresses  the  issue 
of  simulation  standards.  However,  we  were  not  able  to  determine  exactly  what  standardization 
issues  were  addressed  by  the  document  (i.e.,  whether  it  specified  a  system  configuration,  or 
programming  techniques,  or  minimum  simulation  capabilities,  or  some  combination  or  these). 

We  believe  that  AFSPACECOM  has  a  definite  need  for  embedded  training.  Alternative 
means  of  providing  hands-on  training  are  not  yet  widely  available.  Moreover,  AFSPACECOM 
has  an  ongoing  need  for  Mission  Readiness  Training  (MRT)  for  qualified  crewmembers.  The 
systems  operators  need  some  means  of  practicing  and  improving  their  job  skills. 

In  approaching  AFSPACECOM  regarding  embedded  training  research,  it  would  be  best  to 
view  their  problem  in  the  context  of  operational  readiness.  The  Laboratory  should  talk  to 
AFSPACECOM  about  their  need  fo:  an  effective  and  inexpensive  means  of  providing  operator 
readiness  training,  for  which  improved  embedded  training  might  be  one  of  the  possible  solutions. 


"Researchable'  Embedded  Training  Issues 


Some  of  the  user  problems  regarding  embedded  training  potentially  could  be  solved  through 
research.  Other  problems  require  non-research  solutions.  Perhaps  the  most  serious, 

all-pervasive  problem  associated  with  embedded  training  stems  from  lack  of  widespread  user 
awareness  of  the  medium.  Consequently,  potential  users  have  little  interest  in  embedded 

training  as  a  solution  to  their  training  problems.  As  we  have  pointed  out,  even  those 
organizations  who  are  currently  using  embedded  training  often  do  not  call  it  by  that  name. 
Understandably,  there  is  little  or  no  dialogue  among  major  commands  and  organizations  to 
share  information  about  embedded  training  because  no  one  thinks  of  it  as  a  fully  capable 
training  medium.  Embedded  training  is  perceived  as  an  equipment-specific  capability  rather 

than  a  separate  phenomenon  which  can  fit  into  a  total  training  solution. 

The  lack  of  awareness  of  embedded  training  as  an  acceptable  part  of  a  total  training 
solution  might  reflect  the  lack  of  a  single  point  of  focus  for  embedded  training  in  the  Air  Force. 
Unlike  the  Army  and  Navy,  the  Air  Force  has  no  official  directive  which  requires  program 
managers  to  consider  embedded  training  during  the  acquisition  process.  Moreover,  the  Air 
Force  has  no  central  coordinating  agency  to  take  the  lead  in  its  embedded  training  efforts. 
The  other  Services  have  recognized  organizations  with  responsibility  for  information  and  guidance 
regarding  embedded  training.  The  Army  has  the  Program  Manager  for  Training  Devices 
(PM-TRADE)  and  the  Army  Research  Institute  (ARI).  The  Navy  has  the  Naval  Training  Systems 
Center  (NTSC). 

The  consequences  of  this  lack  of  awareness  by  potential  Air  Force  user  agencies  about 
embedded  training  are: 

1 .  , «  failure  to  capitalize  on  the  proven  efficiency  and  overall  quality  of  embedded  training. 

Organizations  must  constantly  re-learn  lessons  and  make  the  same  mistakes  over  again 
because  they  are  not  aware  of  what  they  can  learn  from  others  with  similar  experiences. 
Or,  »hey  simply  fail  to  do  things  that  would  have  been  possible  if  they  had  known  how 
to  plan  properlv  for  the  use  of  embedded  training. 
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2.  A  hampered  ability  to  improve  embedded  training  by  research  efforts.  It  will  be  difficult 
to  get  support  for  an  embedded  training  research  program  because  users  may  not 
immediately  realize  how  it  can  benefit  them.  In  other  words,  it  will  be  difficult  to  make 
embedded  training  better  until  you  get  the  users’  attention  and  the  users  decide  that 
they  need  an  improved  training  capability. 

Until  more  Air  Force  user  organizations  develop  a  broader  understanding  of  embedded 
training,  persuading  them  to  support  embedded  training  research  will  be  very  difficult.  However, 
there  are  some  specific  areas  where  research  may  be  of  value  to  the  customers  we  visited. 
These  areas  are  outlined  and  discussed  below: 

1.  Embedded  Training  System  Architecture 

2.  Assessment  of  Cost  and  Training  Effectiveness  of  Embedded  Training 

3.  Optimal  Features  of  an  Embedded  Training  System 

4.  Role  of  Embedded  Training  in  a  Total  Training  System 

5.  Development  of  Specifications  for  Embedded  Training 


Embedded  Training  System  Architecture 


Embedded  training  systems  often  fail  to  be  implemented  because  of  real  or  perceived 
technical  obstacles.  When  embedded  training  systems  are  implemented,  they  often  fail  to 
incorporate  desirable  training  features.  Poorly  designed  embedded  training  can  fail  to  increase 
readiness  skill  while  reducing  operating  system  performance.  Thus,  embedded  training  designers 
and  developers  would  benefit  from  guidelines  which  help  make  embedded  training  architecture 
more  efficient  and,  perhaps,  more  standardized. 

Embedded  training  designers  and  developers  must  adapt  to  a  variety  of  host  systems. 
Devising  meaningful  guidelines  could  be  difficult.  However,  guidelines  would  provide  two  payoffs. 
First,  users  would  know  what  kind  of  operational  characteristics  they  can  expect  in  the  embedded 
training  for  a  given  class  of  system.  Second,  such  guidelines  would  assure  "safe”  implementations 
of  embedded  training. 

What  kinds  of  guidelines  would  be  useful?  At  the  highest  level,  guidelines  are  needed 
which  provide  the  designers  and  developers  with  a  rule  of  thumb  for  assessing  embedded 
training  costs.  These  guidelines  snould  take  into  account  all  or  most  of  the  risks  associated 
with  adding  an  embedded  training  capability  to  a  system.  They  should  also  take  into  account 
the  varying  levels  of  sophistication  possible  in  embedded  training  scenarios.  More  specific 
guidelines  could  help  System  Program  Office  personnel  to  understand  the  range  of  embedded 
training  features  available  and  the  incremental  cost  and  technical  impact  of  incorporating  these 
features  on  the  system.  For  example,  because  signal  processing  systems  usually  have  a 
built-in  system  test  capability,  adding  a  rudimentary  form  of  embedded  training  (i.e.,  providing 
a  few  practice  scenarios)  is  likely  to  be  a  cheap  and  technologically  feasible  alternative. 
However,  making  "smart"  scenarios,  adding  a  performance  recording  capability,  or  providing  the 
users  with  a  simulation  authoring  capability  might  increase  costs  significantly  and  might  provide 
system  designers  and  developers  with  difficult  technological  problems. 
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Assessment  of  Cost  and  Training  Effectiveness  of  Embedded  Training 


The  people  we  interviewed  could  not  produce  separate  cost  figures  for  the  embedded 
training  component  of  their  systems.  Most  guessed  that  the  addition  or  inclusion  of  the 
embedded  training  capability  had  a  relatively  minor  impact  on  the  overall  cost  of  system 
acquisition  or  development;  however,  they  could  provide  neither  specific  figures  nor  a  percentage 
estimate.  The  inability  of  program  management  personnel  to  assess  the  actual  costs  of 
embedded  training  was  usually  due  to  the  fact  that  the  embedded  training  component  of  the 
system  utilized  the  system’s  built-in  test  capability  (e.g.,  PAVE  PAWS,  BMEWS  and  AWACS). 
Because  embedded  training  costs  are  usually  “included"  in  a  system’s  basic  test  capability, 
costing  embedded  training  separately  could  not  be  done  by  those  program  offices.  If  the 
Laboratory  were  to  become  involved  in  assessing  the  cost  impact  of  embedded  training,  accurate 
cost  figures  could  probably  be  obtained  from  programs  like  the  F-16  OBEWS  which  were 
specifically  designed  as  embedded  training  systems.  All  the  costs  of  such  a  project  would  be 
directly  related  to  the  embedded  training. 

Assessments  of  embedded  training  effectiveness  were  similarly  informal.  Those  training 
effectiveness  assessments  which  we  found  consisted  merely  of  gathering  comments  from  system 
users  in  the  field  and  assessing  subjectively  whether  operator  performance  had  improved.  (It 
might  be  noted  that  everyone  we  talked  to  reported  that  the  embedded  training  capability  of 
the  system  helped  improve  crew  performance.)  We  do  not  doubt  that  embedded  training  has 
had  a  positive  effect  on  crew  performance.  However,  up  to  now  the  extent  to  which  operational 
proficiency  is  improved  by  embedded  training  has  not  been  measured  systematically. 

Most  of  the  user  organizations  that  were  actually  using  embedded  training  in  the  field  were 
not  particularly  disturbed  about  the  lack  of  cost  and  training  effectiveness  data.  They  already 
had  their  systems  in  place  and  had  no  need  to  re-justify  them.  However,  the  lack  of 
effectiveness  data  is  a  problem  for  organizations  like  ASD,  which  are  assessing  whether  or 
not  they  should  add  embedded  training  to  existing  or  future  systems.  It  is  not  as  difficult  a 
decision  for  AFSPACECOM  to  make  regarding  the  inclusion  of  embedded  training  in  a  new 
system,  as  their  systems  have  traditionally  come  with  the  capability  built-in.  ASD  does  not 
have  this  luxury.  ASD  must  decide  whether  to  add  embedded  training,  and  they  need  the 
kind  of  information  necessary  to  make  this  decision. 


Optimal  Features  for  Embedded  Training 


It  is  not  yet  clear  what  functions  or  features  an  optimal  embedded  training  system  ought 
to  have.  Several  attempts  have  been  made  to  specify  what  features  "good"  embedded  training 
should  have.  Little  research  has  related  embedded  training  features  to  specific  job  skills  or 
assessed  the  cost  effectiveness  of  particular  training  features.  The  Army  Research  Institute 
has  suggested  the  following  features  as  representative  of  fully  functional  embedded  training 
(Strasel,  Dyer,  Roth,  Alderman,  &  Finley,  1988): 

1.  Generates  target  or  threat  data; 

2.  Feeds  these  data  into  and  through  the  operational  equipment  to  the  system's  operator(s) 
or  maintainer(s)  by  means  of  their  normal  displays  and  indicators; 

3.  Presents  the  input  data  so  as  to  realistically  depict  what  would  occur  in  an  operational 
exercise  of  the  system  against  a  real  threat; 
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4.  Requires  the  operator(s)  or  maintainer(s)  to  perform  their  normal  and  proper  tasks  and 
duties  in  response  to  the  simulated  mission  inputs; 

5.  Simultaneously  and  interactively  assesses  and  records  the  performance  of  the  operator(s) 
or  maintainer(s)  and  reacts  to  that  performance  as  the  real  threat  would; 

6.  Provides  an  appropriate  level  of  performance  measurement  and  recording  to  allow  both 
individual  feedback  after  a  session  and  semi- permanent  records  of  performance;  and 

7.  Usually  allows  for  the  presentation  of  computer-assisted  or  otherwise  programmed 
instruction  on  related  job-relevant  tasks  and  sub-tasks  in  addition  to  those  which  are 
strictly  operational  mission  performance  tasks. 

ARI’s  set  of  criteria  provides  a  view  of  what  an  "ideal"  embedded  training  system  would 
look  like.  However,  it  does  not  address  the  trade-offs  that  would  need  to  be  made  if  a  “fully 
functional"  embedded  training  system  were  too  expensive,  technologically  too  difficult,  or  not 
needed  to  meet  training  requirements.  For  example,  if  synthetic  threats  have  caused  false 
alarms,  to  what  extent  can  the  threats  be  presented  less  realistically  without  diminishing  the 
value  of  practicing  with  them?  What  aspects  of  an  operator's  performance  need  to  be  recorded? 
Is  supplemental  computer-assisted  instruction  on  a  separate  system  acceptable  or  must  training 
be  limited  to  the  host  system? 

We  have  found  that  embedded  training  features  vary  widely  from  system  to  system.  On-line 
feedback  is  seldom  incorporated.  The  degree  of  inclusion  of  other  features  varies  widely 
among  systems.  For  example,  systems  vary  in  the  degree  to  which  they  allow  scenario 
authoring.  Some  have  a  rewind  capability,  but  not  a  fast  forward  capability.  Others  have  the 
opposite.  Some  have  a  scenario  freeze  capability;  some  do  not.  Moreover,  not  all  embedded 
training  features  are  used  even  if  they  are  available  in  the  system.  For  example,  the  PAVE 
PAWS  embedded  training  system  can  record  the  specific  switch  actions  made  by  a  trainee 
during  scenarios.  However,  this  capability  is  seldom  used  by  AFSPACECOM  to  evaluate  crew 
performance  or  to  provide  immediate  feedback  to  the  trainee.  In  our  opinion,  the  capabilities 
of  many  currently  fielded  embedded  training  systems  only  coincidentally  meet  the  needs  of  the 
operational  units.  There  is  no  hard  evidence  to  suggest  what  features  do  and  do  not  contribute 
to  mission  readiness  or  enhancement. 

ASD  has  expressed  an  interest  in  research  on  the  types  of  real-time  feedback  needed  and 
the  types  of  performance  measurement  needed  for  embedded  training  systems.  Although  other 
organizations  such  as  ESD  and  AFSPACECOM  have  not  expressed  such  interest,  their  use  of 
embedded  training  could  probably  also  benefit  from  research  in  these  areas. 


Role  of  Embedded  Training  in  a  Total  Training  System 


The  role  of  embedded  training  in  the  total  training  picture  for  a  given  system  varies  with 
the  type  of  embedded  training.  With  simulation  ET,  the  embedded  training  is  normally  used 
for  the  practice  or  rehearsal  of  operational  skills.  Other  training  events,  such  as  instruction 
or  feedback,  are  usually  provided  by  supervisors  or  workbooks  on-site.  With  job  aiding,  the 
role  of  the  embedded  training  is  to  provide  assistance  to  the  user  whenever  needed.  With 
tutorial  ET,  the  embedded  training  may  encompass  all  events  of  instruction  from  reminding  the 
user  of  related  background  information  to  presenting  material  to  be  learned  for  the  first  time 
and  testing  the  operator-trainees  retention  and  transfer  of  the  material. 

To  date,  decisions  about  the  role  of  embedded  training  in  the  Air  Force  total  training  system 
have  been  avoided  because  of  a  lack  of  good  information  about  embedded  training.  In 
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organizations  like  ESD  and  SSC,  the  ultimate  “decision"  about  the  inclusion  of  embedded 
training  in  a  system  and  the  specific  features  of  the  embedded  training  has  often  passed  by 
default  to  the  designers  or  developers  of  the  system.  The  users  (e.g.,  AFSPACECQM)  have, 
for  the  most  part,  simply  made  do  with  what  they  were  given.  Embedded  training  has  seldom 
been  a  focus  of  attention  during  the  acquisition  process. 

The  current  approach  to  embedded  training  development  has  worked  more  or  less  successfully 
for  organizations  like  ESD  and  AFSPACECOM.  Over  a  long  period  of  time,  the  users  have 
come  to  expect  practice  capabilities  in  their  operational  systems.  However,  the  current  approach 
is  much  less  helpful  to  organizations  like  TAC,  Strategic  Air  Command  (SAC),  and  Military 
Airlift  Command  (MAC),  which  have  little  or  no  experience  in  using  embedded  training.  The 
addition  of  an  embedded  training  capability  to  a  system  must  result  from  a  deliberate  effort 
by  these  organizations  to  address  their  operational  training  requirements.  We  believe  that  it 
would  be  helpful  to  the  MAJCOMs  and  ASD  to  have  better  information  for  determining  the 
best  applications  for  embedded  training  within  total  training  systems. 


Development  of  Specifications  for  Embedded  Training 

We  have  found  that  specifications  for  major  systems  such  as  PAVE  PAWS,  BMEWS,  or 
AWACS  tend  not  to  address  the  issue  of  embedded  training  because  the  program  office 
personnel  are  engineers  who  have  little  or  no  experience  in  addressing  training  issues.  The 
fact  that  these  systems  include  an  embedded  training  capability  at  all  is  due  to  the  program 
office’s  clear  understanding  of  the  requirements  from  the  user  command  that  operator  personnel 
must  be  able  to  practice  those  skills  which  are  needed  in  emergency  or  wartime  situations. 
More  recent  specifications  (i.e.,  those  for  the  ATF  in  particular)  have  addressed  the  potential 
need  for  embedded  training.  However,  the  requirements  which  could  be  met  with  embedded 
training  are  not  well  defined  in  these  specifications.  Usually,  the  specification  directs  the 
contractor  to  investigate  the  possible  use  of  embedded  training.  From  the  point  of  view  of 
cost  effectiveness  and  training  effectiveness,  this  practice  appears  to  be  very  risky.  Usually, 
the  more  details  that  can  be  included  in  a  specification,  the  more  accurately  costs  can  be 

estimated  and  the  closer  training  results  will  conform  to  Air  Force  expectations. 

We  think  one  reason  for  the  lack  of  detail  about  embedded  training  in  specifications  may 
be  that  acquisition  agencies  are  afraid  of  driving  up  system  costs  with  ” nonessential"  components. 
Acquisition  agencies  currently  have  no  way  of  accurately  specifying  the  parameters  of  an 
embedded  training  component  of  a  system.  They  also  have  no  way  of  ensuring  that  contractors 
or  developers  use  a  logical  and  systematic  method  for  determining  which  training  requirements 
should  be  addressed  by  embedded  training  and  which  should  not.  As  a  result,  they  include 
embedded  training  in  specifications  only  to  the  extent  that  they  feel  it  will  not  impact  system 
costs. 

The  development  of  guidelines  for  embedded  training  specifications  must  follow  from  a 

determination  of  some  acceptable  way(s)  for  embedded  training  to  fit  into  the  total  training 

picture.  This  should  include  the  development  of  guidelines  for  determining:  (a)  what  kind  of 

requirements  embedded  training  could  potentially  address;  (b)  ways  of  determining  the  cost 
impact  of  selecting  embedded  training  as  a  medium;  and  (c)  ways  of  selecting  the  various 
features  of  an  embedded  training  system.  All  of  these  research  issues  need  to  be  addressed 
prior  to  the  development  of  guidelines  for  embedded  training  specifications. 
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IV.  POTENTIAL  ROLES  FOR  AFHRL 


The  Embedded  Training  R&D  Market 


The  previous  chapter  focused  on  the  problems  of  potential  customers  and  tried  to  identify 
problems  that  could  be  "solved"  with  R&D.  This  chapter  will  focus  on  AFHRL’s  relevant 
strengths  vis-a-vis  those  of  other  R&D  organizations. 


Private  Sector  Capabilities  for  Embedded  Training  R&D 


For  the  purpose  of  describing  private  sector  capabilities,  it  is  probably  more  accurate  for 
us  to  think  in  terms  of  markets  from  which  ET-related  R&D  might  emerge,  rather  than  to  view 
embedded  training  R&D  as  an  established  market.  We  were  able  to  identify  four  such  markets: 

1.  Electronic  Systems  Development 

2.  Operations  Training  Hardware 

3.  Training  Systems  and  Services 

4.  Software  for  General  Purpose  Computers 

In  the  following  sections,  we  will  describe  some  representative  companies  from  each  of 
these  markets  which  have  either  developed  embedded  training  systems  or  performed  ET-related 
R&D.  We  will  also  assess  generally  how  companies  from  these  markets  could  contribute  to 
an  embedded  training  R&D  effort. 


Electronic  Systems  Development  Market 


Private  sector  organizations  in  the  Electronic  Systems  Development  market  include  companies 
such  as  Raytheon’s  Electromagnetic  Systems  and  Missile  Systems  Divisions,  Litton  Industries’ 
C3  Systems  Group,  General  Electric's  RCA  Electronic  Systems  Division  and  Grumman. 
Raytheon's  involvement  includes  the  development  of  the  PAVE  PAWS  and  BMEWS  early  warning 
systems,  and  the  Patriot  missile  system.  Litton  developed  Modular  Control  Equipment  (MCE) 
in  a  joint  project  for  the  Marine  Corps  and  the  Air  Force.  RCA  developed  the  AEGIS  shipboard 
missile  weapon  control  system.  Grumman’s  division  in  Melbourne,  Florida,  is  currently  working 
on  the  advanced  radar  systems  on  the  Joint  Surveillance  Target  Attack  Radar  System  (JSTARS). 

The  private  sector  companies  in  the  Electronic  Systems  Development  market  compete  with 
each  other  on  the  basis  of  the  overall  functionality  of  their  systems.  Although  their  systems 
can  usually  be  modified  to  include  embedded  training,  they  generally  do  not  perceive  themselves 
to  be  primarily  developers  of  training  or  of  training  systems.  From  the  perspective  of  embedded 
training  R&D,  much  of  their  strength  lies  in  the  area  of  system  design,  system  development, 
and  the  production  of  large  electronic  systems, 

AFHRL  does  not  compete  in  the  Electronic  Systems  Development  market.  However,  it  may 
be  able  to  draw  upon  the  expertise  of  one  or  more  of  these  companies  in  undertaking  R&D 
problems  related  to  the  design  and  engineering  of  embedded  training  within  large  electronic 
systems. 
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Operations  Training  Hardware  Market 


The  Operations  Training  Hardware  market  is  composed  of  companies  which  produce  equipment 
that  can  be  used  for  operations  training  or  mission  rehearsal  training.  This  market  also  contains 
smaller  sub-groups  of  companies  which  produce  particular  types  of  operations  training  equipment. 
For  example,  both  Loral  Electro-Optical  Systems  and  Saab  Training  Systems  produce  laser 
transmitters  which  can  be  strapped  onto  weapon  systems  to  simulate  the  effects  of  live 
ammunition.  Raytheon’s  Submarine  Signal  Division,  United  Technologies’  Norden  Systems,  and 
United  Industrial  Corporation’s  AAI  all  produce  devices  which  can  be  used  to  inject  signals 
into  shipboard  radar  and  sonar  systems  for  training  purposes.  AAI  has  applied  this  technology 
to  EW  training  aboard  aircraft  via  its  F-16  OBEWS  project. 

Unlike  the  companies  in  the  Electronic  Systems  Development  market,  companies  in  the 
Operations  Training  Hardware  market  openly  compete  for  training  business.  Their  competitive 
edge  in  this  market  results  primarily  from  their  capabilities  to  produce  training  hardware.  These 
companies  have  training  specialists  on  their  staffs,  and  they  are  familiar  with  the  types  of 
documentation  associated  with  the  Instructional  Systems  Development  (ISD)  approach  to  training. 
Decisions  about  the  appropriate  instructional  application  of  their  equipment  are  often  left  to  the 
users. 

Although  it  can  be  said  that  the  companies  in  this  group  are  producers  of  embedded 
training,  they  have  not  made  a  point  of  marketing  themselves  as  such.  Instead,  they  perceive 
themselves  to  be  competing  in  more  limited  markets  to  produce  "on-board  trainers,”  "radar 
stimulators,"  or  "laser  transmitters."  Conceivably,  they  could  market  themselves  as  embedded 
training  experts,  but  they  gain  little  competitive  advantage  at  present  by  using  the  term.  Their 
customers  know  what  kind  of  services  and  products  they  provide. 

AFHRL  could  probably  look  to  the  companies  in  this  market  sector  for  technical  R&D 
solutions  to  training  problems.  F-16  OBEWS,  for  example,  is  a  solution  to  the  problem  of 
providing  EW  training  when  ranges  are  not  available.  Companies  in  this  market  could  be 
valuable  teammates  in  any  AFHRL  endeavor  to  develop  an  embedded  training  testbed  which 
involves  the  development  of  hardware. 


Training  Systems  and  Services 


Companies  in  the  Training  Systems  and  Sen/ices  market  have  tended  to  concentrate  on 
the  study  and  support  of  embedded  training  rather  than  on  the  development  of  full-scale 
systems  which  include  an  embedded  training  component.  This  group  is  primarily  composed 
of  relatively  small  players,  but  some  larger  firms  have  also  gotten  involved. 

Some  of  the  smaller  companies  in  this  market  include  Applied  Sciences  Associates  (a 
subsidiary  of  Analysis  and  Technology),  Hi-Tech  Systems,  Vector  Research,  Eagle  Technology 
(a  subsidiary  of  Logicon),  and  Dynamics  Research  Corporation.  Applied  Sciences  Associates 
(ASA)  produced  the  ARI-sponsored  tri-Service  review  of  embedded  training  components  (Warm, 
Roth,  Sullivan,  &  Bogner  1987).  ASA  and  Hi-Tech  Systems  collaborated  on  a  project  to  assess 
lessons  learned  from  the  application  of  embedded  training  design  guidelines  to  an  emerging 
system,  the  All  Source  Analysis  System/Enemy  Situation  Correlation  Element,  or  ASAS/ENSCE 
(Evans,  Adams,  Simkins,  Aldrich,  Dyer,  &  Narva,  1988).  However,  ASA  is  better  known  for 
developing  the  10-volume  set  of  guidelines  for  implementing  embedded  training  (Finley  et  al., 
1988).  Hi-Tech  Systems  and  Vector  Research  were  also  contributors  to  this  effort. 
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Eagle  Technology’s  efforts  have  included  assessments  of  embedded  training  technology  and 
recommendations  for  embedded  training  R&D  for  both  the  Navy  and  the  Air  Force  (Malehorn, 
1985a,  1985b,  1986).  E-Tech  has  also  developed  the  SPA-25G  Embedded  Training  System 
(SETS)  for  the  Navy’s  SPA-25G  radar  (Jorgensen,  1986,  1987;  Jorgensen,  Hoskin,  &  Manglass, 
1988;  Jorgensen  &  Muse,  1986).  Unlike  the  systems  described  in  the  previous  section  on 
Operations  Training  Hardware,  this  project  was  undertaken  as  a  training  development  effort 
from  the  onset.  Although  software  engineering  skills  were  needed  to  integrate  the  system  and 
develop  its  software,  no  hardware  development  was  involved.  The  training  developed  for  the 
system  was  designed  within  the  ISO  process.  Finally,  E-Tech  has  evaluated  in-place  embedded 
training  systems  to  develop  guidelines  for  new  systems,  specifically  the  AN/BSY-2  Combat 
Control  System  for  SSN-21  submarines  (Hoskin,  Jorgensen,  &  Manglass,  1988). 

Dynamics  Research  Corporation  (DRC)  produced  a  4-volume  set  of  reports  evaluating  the 
potential  for  embedded  training  aboard  tactical  aircraft  for  ASD/XR  (O’Brien  &  Hess,  1988). 
Based  on  their  recommendations,  a  follow-up  study  was  later  conducted  by  the  Fort  Worth 
Division  of  General  Dynamics,  in  which  these  concepts  were  tested  out  in  a  simulator  (Davis 
et  al„  1986;  McNeley  et  al.,  1988). 

To  our  knowledge,  with  the  exception  of  General  Dynamics,  all  of  the  above-mentioned 
companies  performed  their  embedded  training  work  while  still  small  businesses.  All  market 
their  services  in  the  training  systems  and  services  arena,  which  is  where  they  won  their 
embedded  training  contracts;  and  all  utilize  personnel  with  training  expertise. 

AFHRL  could  probably  look  to  the  various  companies  in  this  market  sector  for  general 
training  expertise,  particularly  in  areas  such  as  training  needs  analysis.  Given  their  experience 
with  training,  these  companies  would  probably  view  embedded  training  in  the  context  of  the 
total  training  picture. 

Although  these  companies  can  also  provide  instructional  design  and  development  services 
(as  evidenced  by  E-Tech’s  work  with  SETS),  their  role  in  the  actual  development  of  major 
systems  with  embedded  training  components  is  probably  limited.  Because  these  companies 
do  not  usually  develop  hardware  (i.e.,  operational  equipment),  it  will  be  difficult  for  them  to 
get  involved  on  the  "ground  floor"  of  embedded  training  for  major  systems  development  except 
as  a  team  member  or  subcontractor  with  one  of  the  major  defense  system  development  firms. 
This  is  not  an  unlikely  scenario  for  these  companies.  However,  due  to  the  fact  that  embedded 
training  tends  to  be  perceived  as  a  practice  medium  currently,  rather  than  as  an  instructional 
medium,  there  may  not  be  a  widespread  demand  for  instructional  development  expertise. 


Software  for  General  Purpose  Computers  Market 


For  the  most  part,  the  literature  on  embedded  training  systems  in  the  military  does  not 
recognize  training  that  occurs  on  general  purpose  computer  systems  to  be  embedded  training. 
This  type  of  training  is  usually  considered  to  fall  under  the  rubric  of  computer-based  training 
or  on-line  help.  However,  if  this  type  of  training  were  included  in  the  definition  of  embedded 
training,  there  are  a  number  of  companies  which  could  potentially  become  involved. 

There  are  three  avenues  by  which  companies  could  participate  in  embedded  training 
development  in  this  market.  The  first  is  by  developing  computer-based  training  or  job  aiding 
to  accompany  software  applications.  The  second  is  by  providing  some  form  of  programming 
or  authoring  capability  so  that  users  themselves  can  develop  training  if  they  wish.  Both  of 
these  approaches  are  fairly  conventional  and  well  explored.  The  third  is  by  concentrating  on 
developing  innovative  solutions  or  approaches  to  training  on  these  systems.  Generally,  this 
involves  the  use  of  Artificial  Intelligence  (Al)  programming  techniques. 
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Major  Government  contractors  which  develop  training  or  provide  job  aiding  for  the  applications 
they  produce  include  UNISYS,  Zenith,  Computer  Sciences  Corporation,  and  IBM  among  others. 
There  are  also  a  number  of  companies  which  primarily  market  their  CBT  development  services; 
these  companies  could  be  contracted  to  develop  training  for  an  existing  application.  These 
companies  include  Flight  Safety  International,  CAE-Link’s  Allen  Corporation  and  McDonnell 
Douglas  Computer  Based  Training  Systems.  Several  of  the  companies  in  the  Training  Systems 
and  Services  market  are  also  able  to  provide  CBT. 

Companies  which  provide  programming  or  authoring  capabilities  with  their  systems  include 
UNISYS,  Honeywell,  Wang,  and  IBM.  Generally,  these  capabilities  are  included  with  larger, 
mainframe  systems;  and  generally,  they  allow  users  to  produce  either  CBT  lessons  or  help 
screens.  However,  the  programming  or  authoring  capabilities  are  usually  much  more  limited 
than  what  could  be  found  in  a  full-blown  programming  language  or  CBT  authoring  package. 

Some  organizations,  with  experience  in  providing  specialized  software  include  Lockheed’s  Al 
Center;  Bolt,  Beranek,  and  Newman  Laboratories,  Inc.  (BBN);  the  University  of  Central  Florida’s 
Institute  for  Simulation  and  Training;  and  Southwest  Research  Institute.  All  of  the  above-listed 
companies  have  explicitly  indicated  an  interest  in  embedded  training  R&D.  BBN  has  already 
produced  an  embedded  training  technology  survey  for  ARI  (Massey,  Harris,  Downes-Martin,  & 
Kurland,  1986).  BBN  has  also  been  involved  with  intelligent  tutoring  systems  development 
since  the  mid-1970s  (Collins,  Warnock,  &  Passafiume,  1975). 

AFHRL  could  look  to  any  of  the  companies  in  this  market  sector  for  both  software  engineering 
and  training  development  skills.  However,  the  companies  which  market  innovation,  rather  than 
systems  or  training  packages,  are  the  most  likely  partners  for  embedded  training  R&D  in 
support  of  the  SSC’s  needs. 


Multi-Market  Competitors 


There  are  some  companies  which  are  so  large  and/or  multi-faceted  that  they  could  compete 
in  several  of  the  above  markets  or,  conceivably,  consolidate  some  of  their  resources  so  that 
they  can  compete  in  the  embedded  training  market  in  a  more  comprehensive  way.  Companies 
like  Boeing,  Lockheed,  IBM  and  McDonnell  Douglas,  for  example,  have  divisions  for  training, 
electronics,  and  aircraft.  Although  these  divisions  are  often  managed  independently,  these 
companies  could  in  theory  marshall  their  resources  to  present  a  total  systems  approach  to  the 
development  of  embedded  training. 

Many  such  companies  are  interested  in  becoming  more  involved  in  embedded  training 
research  and  development.  Boeing  Military  Airplane's  Simulation  and  Training  Group,  for 
example,  recently  completed  an  Independent  Research  and  Development  (IR&D)  project  which 

proposed  an  embedded  training  capability  on  current  and  future  weapon  systems  (Leishman, 
1989).  Both  McDonnell  Douglas  and  Ball  Systems  are  currently  involved  in  the  development 
and  testing  of  the  F-15  On-Board  Simulator  (F-"^  OBS)  for  the  Flight  Dynamics  Laboratory 
(WRDC/FIGX).  Logicon  was  involved  in  the  development  of  an  embedded  training  component 
for  the  CINC-PACAF  Integrated  Decision  Support  System  (CIDSS).  Of  course,  the  two  ATF 
teams  (General  Dynamics/Lockheed/Boeing  and  Northrop/McDonnell  Douglas)  are  also  capable 
of  marshalling  the  necessary  capabilities  to  provide  comprehensive  embedded  training  R&D  for 
that  aircraft. 

At  present,  these  companies'  potential  for  embedded  training  involvement  remains  just 

that-potential.  Most  appear  to  be  waiting  for  the  Air  Force  to  sanction  the  use  of  embedded 

training.  At  the  recent  Interservice/Industry  Training  Systems  Conference,  McDonnell  Douglas 
and  Ball  Systems  company  representatives  indicated  that  although  they  were  interested  in 

embedded  training  development,  they  could  not  do  much  until  the  Air  Force  begins  to  request 
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it.  If  and  when  the  Air  Force  begins  to  encourage  the  use  of  embedded  training,  such 
companies  certainly  have  the  capabilities  in-house  to  bring  to  bear  on  a  variety  of  aspects  of 
embedded  training  R&D. 

Depending  on  one’s  viewpoint,  embedded  training  can  be  considered  either  an  old  or  a 
new  technology.  Companies  which  develop  electronic  systems  have  been  developing  a  form 
of  embedded  training  for  years.  But  embedded  training  itself  has  only  just  begun  to  catch 
the  attention  of  training  specialists.  It  is  not  inconceivable  that  in  some  larger  companies, 
one  division  could  be  trying  to  break  into  the  embedded  training  market,  while  another  has 
been  developing  it  for  years. 

As  for  active,  conscious  involvement  in  embedded  training  R&D,  no  one  company  seems 
to  be  the  market  leader.  Most  private  sector  companies  have  worked  on  a  single  project  or 
perhaps  a  few  related  projects.  Only  a  few  small  companies,  such  as  Applied  Sciences 
Associates  and  Eagle  Technology,  have  made  a  concerted  effort  to  develop  and  promote  their 
"embedded  training"  expertise. 


The  DoD  Embedded  Training  R&D  Agenda 


AFHRL  is  not  the  only  Government  organization  looking  at  embedded  training.  The  Joint 
Technical  Coordinating  Group  for  Training  Systems  and  Devices  (JTCG-TSD)  has  established 
a  tri-Service  working  group  to  coordinate  and  oversee  embedded  training  within  the  Services. 
JTCG-TSD  has  proposed  an  embedded  training  Study  Plan  (1989)  which  is  oriented  around 
five  major  embedded  training  R&D  tasks.  JTCG-TSD’s  first  task  is  to  "evaluate  effectiveness 
of  in-place  embedded  training  systems  (air/surface/submarine)  including  maintenance  trainers." 
Tri-Service  research  efforts  in  support  of  this  task  include  a  joint  ARI/PM  TRADE  effort, 
Tri-Service  Review  of  Existing  System  Embedded  Training  Components  (Warm  et  al.,  1987) 
and  an  NTSC  effort  to  evaluate  embedded  training  for  sub-sea  systems  with  a  view  toward 
application  to  AN/BSY-2  training  system  design  (Hoskin  et  al.,  1988).  The  reports  from  these 
two  projects  do  an  excellent  job  of  describing  and  evaluating  the  systems  they  cover.  However, 
their  work  is  of  limited  relevance  to  Air  Force  systems.  The  ARI/PM  TRADE  study  covers 
three  Air  Force  systems:  WWMCSS,  F-15  OBS,  and  AW  ACS.  The  NTSC  AN/BSY-2  study 
focuses  on  undersea  systems.  Also,  the  JTCG-TSD  effort  has  tended  to  look  to  the  Air  Force 
for  embedded  training  R&D  which  is  specifically  related  to  airborne  systems  (more  particularly 
pilot  training)  when,  in  fact,  the  Air  Force  has  been  regularly  employing  embedded  training  in 
its  large  ground-based  sensor  systems  since  the  1950s.  An  AFHRL  effort  to  evaluate  existing 
Air  Force  embedded  training  systems  could  enhance  the  tri-Service  effort  by  contributing  useful 
data  about  these  systems  (particularly  ground-based  systems),  and  by  providing  more 
system-specific  data  for  the  systems  being  evaluated.  Air  Force  ground-based  radar  systems 
are  in  many  ways  quite  similar  to  Navy  and  Army  radar  systems.  Cross-over  lessons  about 
embedded  training  could  be  learned  by  each  Service  from  studies  of  these  systems. 

The  expressed  purpose  of  the  JTCG-TSD's  second  task  is  to  investigate  selected  tri-Service 
classes  or  sub  classes  of  materiel  items  to  determine  problems,  constraints  and  impediments 
to  embedded  training  implementation,  and  to  analyze  the  resulting  data  to  define  potential 
research  programs  or  administrative  action  that  might  develop  solutions  to  these  problems. 
Tri-Service  efforts  which  support  this  task  include  two  Air  Force  projects,  and  several  other 
Army  and  Navy  projects.  The  Air  Force  projects  are  the  completed  ASD/XR  project  Embedded 
Training  Concepts  for  Tactical  Aircraft  (O’Brien  &  Hess,  1988),  and  a  WRDC/FIGX  project 
currently  in  progress  involving  the  Air  Combat  Engagement  System.  The  Army  projects  were 
the  ARI  10-volume  study,  Guidelines  for  Implementing  Embedded  Training  Systems  (Finley 
et  al.,  1988),  and  a  PM-TRADE  effort  to  derive  embedded  training  system  concepts  and 
strategies  for  the  Armored  Family  of  Vehicles.  The  Navy  projects  consist  of  an  NTSC  concept 
demonstration  effort  to  develop  embedded  training  for  the  SPA-25G  radar  repeater;  another 
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NTSC  effort  to  identify  training  technologies  which  can  be  incorporated  in  embedded  training 
systems;  and  an  effort  to  develop  the  Battle  Force  Research  Simulator  (BFRS),  a  testbed  for 
shipboard  embedded  training  systems  and  concepts. 

The  JTCG-TSD's  third  research  task  is  to  develop  a  centralized  embedded  training  database. 
According  to  the  Study  Plan,  the  performing  agency  and  funding  source  for  this  task  have  yet 

to  be  identified.  The  task  would  involve  more  than  simply  collecting  information  from  the 

Services;  it  would  also  require  defining  who  the  users  would  be  and  their  information  needs, 

selecting  appropriate  hardware  and  software,  and  finding  a  host  organization  to  house  the 

database  and  staff  the  facility. 

JTCG-TSD’s  fourth  task  is  to  conduct  a  case  study  of  potential  embedded  training  applications 
for  Unmanned  Aerial  Vehicles  (UAVs).  The  performing  agency  and  funding  source  for  this 

task  have  not  yet  been  identified,  and  the  task  itself  appears  to  be  in  the  definition  stage  at 
the  moment.  It  would  appear  that  the  task  is  oriented  solely  toward  the  Army. 

Finally,  JTCG-TSD’s  fifth  task  is  to  "develop  guidelines  and  exemplar  products  for  incorporating 
embedded  training  requirements  in  System  Requests  For  Proposal  (RFPs)  and  for  evaluating 
proposals  under  the  Proposal  Evaluation  Plan  (PEP)."  Two  ARI  projects  support  this  task. 
One  is  a  completed  ARI  effort  to  develop  embedded  training  guidelines  for  the  Armored  Family 
of  Vehicles;  the  other  is  an  ongoing  ARI  effort  to  take  these  Armored  Family  of  Vehicles 

guidelines  through  the  Proof-of-Principle  phase  to  the  development  of  Concept  Formulation 

Packages  for  initial  prime-item  Heavy  Forces  Modernization  Program  implementation. 

The  Joint-Services  agenda  is  oriented  primarily  toward  the  Army  and  the  Navy.  It  does 
not  address  adequately  the  needs  of  the  Air  Force  customers  we  identified  in  Chapter  III. 


AFHRL's  Special  Strengths  Vis-a-Vis  Embedded  Training  R&D 


No  one  has  taken  the  lead  in  conducting  embedded  training  R&D  for  the  Air  Force.  AFHRL’s 
strengths  would  better  suit  it  for  some  embedded  training  research  roles  than  for  others. 
AFHRL's  organizational  strengths  with  regard  to  embedded  training  R&D  are,  first,  its  natural 
competitive  advantages  as  an  Air  Force  R&D  organization;  and  second,  the  resources  and 
talent  it  already  has  on  board  which  can  be  brought  to  bear  on  embedded  training  R&D. 
We  believe  some  of  AFHRL’s  relevant  advantages  to  be  as  follows: 

1.  Existing  Contacts  with  Potential  Embedded  Training  R&D  Customers 

2.  Job  Analysis  Expertise 

3.  Specific  Training  Expertise 

4.  System  Design  Expertise 


Existing  Contacts  with  Potential  Embedded  Training  R&D  Customers 


As  previously  mentioned,  AFHRL  already  has  established  close  working  relationships  with 
several  potential  customers  via  its  ongoing  programs.  Few  other  organizations  can  match 
AFHRL’s  extensive  Air  Force  customer  base.  Several  AFHRL  customers  have  a  high  degree 
of  potential  for  supporting  embedded  training  research  (e.g.,  AFSPACECOM,  ASD,  ESD,  Pacific 
Air  Forces  (PACAF),  and  the  major  Continental  U.S.  (CONUS)  flying  commands).  These 
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existing  contacts  give  AFHRL  an  edge  over  other  R&D  organizations  in  terms  of  established 
relationships  with  potential  customers. 

AFSPACECOM  is  an  important  potential  customer  because  the  command  uses  embedded 
training  in  its  large-scale  radar  systems.  The  Logistics  and  Human  Factors  Division  (AFHRL/LR) 
has  had  a  close  working  relationship  on  several  projects  with  AFSPACECOM.  In  addition, 
AFSPACECOM  has  formally  requested  AFHRL's  assistance  fu  C2  training  research.  The 
Ground  Operations  Branch  (AFHRLVLRG)  is  currently  engaged  in  several  projects  related  to  C2 
Team  Training,  including  development  of  a  C2  part-task  trainer  for  AFSPACECOM.  AFHRLVLRG 
also  participates  with  ESD  in  the  Strategic  Defense  Initiative  (SDI)  and  Battle 

Management/Command,  Control,  and  Communications  (BM/C3)  programs.  AFSPACECOM  and 

ESD  could  identify  issues  on  which  AFHRL/LR  could  perform  embedded  training  research. 

Both  the  Manpower  and  Personnel  Division  (AFHRLVMO)  and  the  Operations  Training  Division 
(AFHRLVOT)  have  previously  established  contacts  at  ASD,  TAC,  SAC,  and  MAC.  AFHRL/MO’s 
support  for  these  customers  includes  job/task  analysis.  AFHRL/OT’s  support  of  its  customers 
consists  primarily  of  conducting  aircrew  training  research;  disseminating  its  accumulated  findings 
and  expertise  to  users;  and,  development  of  new  technologies  for  aircrew  training.  The 
involvement  of  these  two  divisions  provides  AFHRL  with  a  comprehensive  view  of  aircrew 
training  and  a  full  spectrum  of  training  contacts  at  ASD  and  the  flying  MAJCOMs. 

Most  of  the  Training  Systems  Division’s  (AFHRLVID)  current  contacts  do  not  appear  to  be 
potential  embedded  training  research  customers.  The  Air  Training  Command,  an  important 
AFHRLVID  customer,  does  not  appear  to  be  a  potential  embedded  training  R&D  customer.  ATC 

deals  primarily  with  training  initial  skills,  which  tend  to  utilize  dedicated  training  equipment  or 

simulators  rather  than  actual  equipment;  embedded  training,  by  definition,  requires  the  use  of 
actual  operational  equipment. 


Job  Analysis  Expertise 


AFHRL  also  has  an  edge  over  potential  private  sector  competitors  insofar  as  its  previous 
work  with  such  a  wide  range  of  Air  Force  customers  has  given  AFHRL  insight  into  the  training 
needs  of  particular  job  categories.  Some  of  these  job  categories,  such  as  Command  and 
Control  (C2)  systems  operations  and  aircrew  positions,  have  a  high  potential  for  the  use  of 
embedded  training,  because  of  their  requirement  for  signal  processing  skills.  Several  AFHRL 
divisions  have  worked  extensively  with  one  or  more  of  these  job  categories,  and  are  therefore 
quite  familiar  with  the  training  requirements  for  these  positions. 

Knowledge  of  the  skills  required  for  specific  occupational  specialties  provides  AFHRL  an 
advantage  in  working  on  embedded  training  problems.  Because  one  of  the  primary  problems 
in  training  in  general,  including  embedded  training,  is  related  to  the  selection  of  the  proper 
media,  AFHRL’s  ability  to  identify  these  requirements  for  numerous  Air  Force  specialties  (AFSs) 
and  match  an  appropriate  medium  with  them  to  achieve  training  effectiveness  and  cost 
effectiveness  will  give  it  a  distinct  advantage  over  the  competition. 

For  the  purposes  of  embedded  training  R&D,  AFHRL's  familiarity  with  aircrew  training  (via 
AFHRLVOT)  and  Command  and  Control  team  training  (via  AFHRLVLR)  is  quite  relevant.  AFHRLVLR 
has  an  excellent  understanding  of  C2  job  skills  as  a  result  of  its  research  with  the  C2  Team 
Training  project.  This  research  has  enabled  AFHRL/LR  to  build  up  expertise  in  task  analysis, 
performance  assessment,  and  trainer  development  for  C2  skills.  All  of  these  skills  would  be 
potentially  transferable  to  C2  embedded  training  research. 

Both  AFHRLVMO  and  AFHRL/ID  have  had  extensive  exposure  to  training  requirements  for 
maintenance  specialties.  AFHRLVMO  has  tended  to  focus  on  the  analysis  of  maintenance  job 
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skills,  whereas  AFHRLVID’s  R&D  efforts  have  centered  on  the  development  of  systems  and 
tools  to  support  training  development,  such  as  the  Training  Decisions  System  (TDS),  Job 
Performance  Measurement  and  Training  Assessment  procedures,  and  development  of  advanced 
instructional  technologies  such  as  Intelligent  Tutoring  Systems. 


Training  Expertise 


MO  has  accumulated  a  database  containing  Manpower,  Personnel,  and  Training  requirements 
data  for  existing  Air  Force  weapon  systems  while  in  the  process  of  developing  tools  for 
forecasting  life-cycle  costs  (LCC)  of  new  technologies.  As  information  about  the  cost  and  use 
of  embedded  training  becomes  available,  it  could  be  incorporated  into  this  database.  AFHRL/MO 
may  be  able  to  develop  a  similar  model  to  estimate  what  impact  adding  embedded  training 
has  on  the  overall  training  and  LCC  for  a  given  system.  This  new  model  would  enhance  the 
existing  AFHRL/MO  LCC  models. 

AFHRL/ID  has  several  programs  which  could  potentially  be  extended  to  embrace  embedded 
training  R&D.  These  include  the  Job  Performance  Measurement  (JPM)  R&D  program,  the 
Training  Decisions  System  (TDS),  and  AFHRUID's  work  on  Intelligent  Tutoring  Systems  (ITS) 
development.  The  JPM  project,  which  is  designed  to  develop  procedures  for  selecting, 
developing,  administering,  and  interpreting  measures  of  training  effectiveness,  could  conceivably 
be  augmented  to  include  embedded  training.  The  TDS,  which  helps  users  evaluate  the  costs 
and  benefits  of  various  training  alternatives,  could  easily  be  expanded  to  include  embedded 
training  as  another  training  alternative  to  be  considered  in  the  Air  Force  concept  of  total 
training.  AFHRL/ID’s  work  in  ITS  could  be  the  basis  for  a  project  oriented  toward  applying 
Al  techniques  to  embedded  training,  such  as  the  development  of  intelligent,  context-sensitive 
job  aids  to  accompany  applications  software  packages. 

AFHRL/LR’s  programs  on  C2  training  could  very  well  be  augmented  to  include  an  investigation 
of  embedded  training.  At  present,  the  Ground  Operations  Branch  (AFHRL/LRG)  is  involved 
with  several  projects  for  AFSPACECOM,  PACAF,  and  several  other  users.  Most  of  these 
projects  are  small  and  oriented  toward  development  of  a  single  training  system.  The  projects 
include  a  part-task  trainer  for  AFSPACECOM,  a  prototype  training  system  for  PACAF,  another 
prototype  training  system  for  Modular  Control  Equipment  (MCE)  teams  for  ESD,  and  a  prototype 
training  system  for  Logistics  C2.  All  of  these  projects  represent  prototypical,  dedicated  training 
systems  and  do  not  include  embedded  training. 

Finally,  AFHRL/OT’s  efforts  to  produce  design  guidelines  and  specifications  for  aircrew 
training  systems  could  be  augmented  to  include  the  development  of  guidelines  and  model 
specifications  for  embedded  training.  At  present,  AFHRLyOT’s  efforts  have  centered  on  gathering 
data  related  to  the  improvement  of  aircrew  training. 


Systems  Design  Expertise 


AFHRL/LR’s  work  on  Computer-Aided  Design/Computer-Aided  Engineering  (CAD/CAE)  involves 
the  development  of  tools  that  enable  engineers  to  consider  the  reliability,  availability,  and 
maintainability  trade-offs  they  would  be  making  in  adopting  particular  designs.  With  the  right 
kind  of  data  about  embedded  training  system  architecture,  the  same  kinds  of  tools  might  be 
extended  to  assess  the  impact  of  adding  an  embedded  training  component  to  a  system. 
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Where  AFHRL  Could  Make  Important  and  Unique  Contributions 


If  AFHRL  becomes  involved  in  embedded  training  R&D,  it  would  have  a  competitive  edge 
in  addressing  the  following  R&D  issues.  These  are  areas  in  which  AFHRL  has  unique  strengths: 

1.  Job  Performance  Measurement.  AFHRL  could  apply  its  accumulated  knowledge  of  job 
performance  measurement  to  the  task  of  identifying  what  performance  measures  are 
needed  for  embedded  training. 

2.  Manpower,  Personnel  and  Training  (MPT)  Integration.  MPT  integration  technologies 
could  be  modified  to  reflect  the  impact  of  the  use  of  embedded  training  on  the  life-cycle 
costs  of  a  system. 

3.  Concurrent  Engineering.  AFHRL  could  build  on  rts  experience  with  the  development 

of  Reliability,  Availability,  and  Maintainability  in  Computer-Aided  Design  (RAMCAD)  and 

ergonomics  (CREW  CHIEF)  to  develop  similar  tools  'or  embedded  training  design 
considerations  and  trade-offs. 

4.  Investigation  of  the  Feasibility  of  Embedded  Training  for  Specific  Job  Co.egories.  AFHRL 
is  particularly  well  equipped  to  explore  implementations  of  embedded  training  in  the 
areas  of  Command  and  Control,  aircrew  operations,  and  maintenance. 

5.  Applications  of  Artificial  Intelligence  (Al)  to  Training.  AFHRL  has  had  extensive  experience 
in  cooperative  efforts  to  apply  Al  to  training.  This  experience  could  well  be  applied 
to  the  development  of  solutions  to  the  problem  of  training  the  users  of  applications 
software. 

AFHRL  should  undertake  R&D  in  the  following  areas  only  with  the  assistance  of  experienced 
private  sector  firms  or  other  DoD  agencies  which  have  particular  strengths  upon  which  AFHRL 
could  capitalize.  These  are  areas  for  which  either  (a)  AFHRL  has  some  experience,  but  not 
necessarily  unique  strengths;  or  (b)  AFHRL  can  profitably  team  with  others  because  of  the 

advantage  it  would  bring: 

1.  Embedded  Training  Design  Issues.  These  are  issues  which  pertain  to  determining  the 
form  and  features  of  embedded  training,  but  which  stop  short  of  actually  producing 
hardware/software.  These  issues  may  include  R&D  on  safe  implementations  of  embedded 
training  for  aircraft;  determination  of  whether  or  not  standardization  of  embedded  training 
features  is  possible  and/or  desirable;  and  assessment  of  the  incremental 
economic/instructional  benefits  of  adding  particular  features  to  embedded  training  systems. 

2.  Definition  of  the  Role  of  Embedded  Training  in  a  Total  Training  System.  Although 

AFHRL  does  have  strengths  in  terms  of  its  ability  to  research  Total  Training  Systems, 
other  Air  Force  organizations,  such  as  ASD  and  ATC,  are  also  heavily  involved  in  this 

area.  Moreover,  the  other  Services  have  embedded  training  researchers  who  have 

been  working  with  this  issue  for  some  time.  It  would  be  advisable  for  AFHRL  to 
pursue  a  joint  approach  to  this  problem. 

3.  Development  of  Model  Specifications  for  Embedded  Training.  Again,  although  AFHRL 
has  considerable  experience  in  the  development  of  model  specifications,  it  could  benefit 
from  cooperative  efforts  with  other  Air  Force  organizations  and  the  other  Services  in 
this  area. 

AFHRL  should  avoid  undertaking  R&D  in  the  following  area  because  of  a  lack  of  expertise 
and  the  high  risk  associated  with  such  R&D: 
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1.  Hardware  or  Software  Development.  AFHRL  is  not  we  i  equipped  to  take  on  the 
development  o*  embedded  training  hardware  or  software.  Whenever  possible,  it  should 
try  to  perform  research  on  systems  being  developed  or  acquired  by  its  customers.  For 
example,  WRDC/FIGX  and  ASD/ENE  previously  requested  AFHRL  to  conduct  embedded 
training  R&D  with  the  Air  Combat  Engagement  System  (ACES). 


V.  RECOMMENDED  IMPLEMENTATION  STRATEGIES 


Potential  Technology  Development  Areas 

Chapters  II  and  III  discussed  trends  in  the  use  of  embedded  training  and  potential  users’ 
needs.  Chapter  IV  looked  at  potential  strengths  that  AFHRL  could  bring  to  embedded  training 
R&D.  This  chapter  identifies  a  set  of  R&D  issues  which  are  derived  from  significant  user 
needs,  and  which  AFHRL  is  well  suited  to  address.  These  issues  could  form  a  potential 

embedded  training  R&D  agenda  for  AFHRL.  We  suggest  neither  that  this  agenda  is  complete 

nor  that  all  the  issues  have  to  be  addressed  by  AFHRL.  For  some  of  the  issues  on  the 
agenda,  the  risk  may  be  too  great  or  the  payoff  too  small.  However,  we  feel  that  any 
embedded  training  research  implementation  strategy  would  address  some  of  these  issues. 

In  each  of  the  following  sections  we  will  outline  an  R&D  issue  which  AFHRL  could  investigate. 

Each  R&D  issue  is  discussed  in  terms  of  the  technology  development  area,  the  potential 

customers,  the  relevant  AFHRL  strengths,  and  the  risks  involved. 


The  Safety  of  Embedded  Training  Systems 


Technology  Development  Area :  On  the  basis  of  their  existing  working  relationship  with 
AFSPACECOM  and  the  high  concentration  of  embedded  training  systems  at  AFSPACECOM, 
AFHRL/LR  could  take  the  lead  in  a  study  for  AFSPACECOM.  This  effort  might  possibly  be 
combined  with  some  other  new  or  ongoing  AFHRL  effort  with  AFSPACECOM,  such  as  a  project 
to  assess  alternatives  for  maintaining  operational  readiness.  Such  a  study  would  describe  in 
detail  how  the  embedded  training  for  a  given  system  interfaces  with  the  system  hardware  and 
software,  how  simulated  s:]nals  enter  the  system,  what  types  of  operator  switch  actions  are 
allowable,  and  the  "smartness"  of  simulated  targets.  It  could  also  assess  the  long-term  effects 
of  embedded  training  on  the  host  system;  the  effect  of  and  potential  for  adding  features  to 
the  embedded  training  component;  and  user  embedded  training  requirements. 

ASD  also  has  concerns  about  the  safety  of  embedded  training.  AFHRI70T  could  work 
with  ASD  to  assess  safety  features  required  for  airborne  embedded  training.  A  pilot  test  project 
using  AFHRL/OT’s  simulators  could  be  the  first  step  in  assessing  the  safety  factors  required 
for  embedded  training.  Some  of  the  key  issues  to  concentrate  on  are  those  listed  by  ASD/ENE 
(see  Chapter  III).  Such  a  project  would  also  contribute  to  the  JTCG-TSD  embedded  training 
agenda. 

Potential  Customer:  Both  ASD  and  AFSPACECOM  have  expressed  concerns  about  the 
impact  of  embedded  traininq  on  the  safety,  security,  and  performance  of  their  systems.  Some 
managers  are  consequently  reluctant  to  use  embedded  training.  ASD  is  expressing  the  concerns 
of  the  flying  community,  some  of  whom  feel  that  any  aircraft  component  which  does  rot  directly 
contribute  to  the  operational  mission  is  an  added  weight  or  burden  to  the  aircraft  and  a 
potential  safety  hazard.  This  is  an  important  pioblem  because  concerns  about  the  safe 
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implementation  of  embedded  training  are  currently  inhibiting  AFSPACECOM  and  ASD  in  their 
use  of  embedded  training. 

Relevant  AFHRL  Strengths:  AFHRIVLRL’s  efforts  to  incorporate  maintenance  and  logistics 
considerations  into  the  hardware  design  process  on  such  projects  as  the  Ground-Launched 
Cruise  Missile,  the  F-15E,  and  the  Integrated  Electronic  Warfare  System  have  enabled  it  to 
develop  a  certain  degree  of  in-house  software  engineering  expertise  which  could  be  used  in 
projects  of  this  type.  To  the  extent  that  simulation  equipment  mirrors  operational  equipment 
design,  AFHRL's  Technology  Development  Branch  (AFHRL/OTE)  also  has  the  appropriate 
engineering  experience  to  initiate  an  embedded  training  research  project  involving  safe  embedded 
training  for  aircraft.  AFHRiyOTE’s  perspective  on  aircraft  training  simulators  should  also  be 
useful  in  recognizing  and  addressing  the  training  issues  involved  in  such  research. 

Risks:  Although  it  has  some  engineering  experience,  AFHRL  lacks  the  in-depth  technical 
expertise  in  software  engineering  to  conduct  R&D  involving  the  safe  implementation  of  embedded 
training  without  some  outside  assistance.  A  contractor  with  sufficient  software  engineering 
expertise  would  reduce  the  technical  risks  of  this  project. 


Intelligent  Tutors  for  Standard  Systems 


Technology  Development  Area:  Applications  software  is  in  use  today  in  standard  computer 
systems  throughout  the  Air  Force.  There  is  a  continuing  need  to  identify  specific  user  problems 
with  the  applications  package  and  to  train  the  users.  Our  site  visits  indicated  that  no  amount 
of  user  friendliness  in  an  applications  package  can  match  having  an  expert  user  nearby  for 
assistance.  Providing  some  type  of  intelligent  job  aids  for  applications  software  may  be  a 
potential  solution. 

The  Laboratory  could  take  the  lead  in  supporting  the  SSC  by  developing  job  aiding  software 
whic.  could  operate  on  the  new  Desktop  III  computers.  This  software  might  consist  of  some 
sort  of  expert  shell  which  would  allow  applications  software  developers  to  create  job  aids  for 
their  users,  and  users  to  query  the  system  to  solve  problems  with  the  software. 

The  same  kinds  of  Al  applications  which  are  developed  to  make  the  development  of 
embedded  training  easier  could  also  be  used  to  create  an  embedded  monitoring  system  for 
software  applications.  Such  a  monitoring  system  could  check  the  user's  ability  to  use  all  of 
the  software  features;  it  could  monitor  the  user’s  need  for  specific  help  during  an  application, 
diagnose  the  specific  problem,  and  prescribe  training  or  remediation;  and,  it  could  provide 
"coaching"  to  the  user. 

Potential  User  Needs:  The  potential  payoffs  could  be  elimination  of  the  cost  of  classroom 
training  for  software  users,  improved  fixes  to  remedy  difficult  user  interfaces,  and  time  savings 
for  innumerable  users.  Both  the  SSC  and  user  commands  should  realize  the  benefit  of  research 
to  eliminate  or  reduce  the  amount  of  user  training  required  for  applications  software.  The 

SSC  program  managers  also  need  assistance  in  identifying  those  software  programs  which 
could  benefit  from  an  embedded  training  capability.  Their  ability  to  specify  exactly  what  form 
the  required  embedded  training  should  take  is  minimal  without  a  methodology  or  decision  aid 
for  embedded  training. 

A  user  like  the  Accounting  and  Finance  Center,  which  uses  numerous  software  programs 
for  financial  applications  on  standard  computer  systems,  would  benefit  from  embedded  training 

for  these  applications.  The  Accounting  and  Finance  Center,  as  well  as  the  entire  Air  Force 

Comptroller  community,  spends  significant  amounts  of  time  training  on-the-job  for  financial 

software  applications.  Ways  of  reducing  these  hours  and/or  increasing  performance  capabilities 
would  be  viewed  as  beneficial  to  the  Air  Force. 
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Relevant  AFHRL  Strengths:  AFHRL's  Intelligent  Systems  Branch  (AFHRL/IDI)  is  at  the 
forefront  of  Air  Force  research  in  the  application  of  Al  to  training.  The  branch  is  currently 
involved  in  over  20  Intelligent  Tutoring  System  (ITS)  projects,  nnd  they  maintain  contacts  with 
many  of  the  leading  researchers  in  the  field.  Among  the  ITS  projects,  the  work  on  intelligent 
"microworlds"  and  rapid  instructional  development  tools  for  ITSs  (such  as  the  Intelligent 
Computer- Assisted  Training  Testbeds)  are  particularly  pertinent  to  the  SSC  and  the  Accounting 
and  Finance  Center. 

Risks:  The  initial  research  required  to  develop  solutions  in  this  area  will  almost  certainly 
be  costly  because  of  the  rapidly  developing  state  of  the  art  in  Al  applications.  Even  if 
sufficiently  intelligent  job  aids  can  be  developed,  developing  them  for  individual  software 
packages  may  add  too  much  to  the  cost  of  the  software.  Alternative  approaches  to  this 
problem  may  focus  on  the  issue  of  developing  a  "generic"  Al  shell  which  can  be  applied  across 
software  applications,  or  developing  a  methodology  for  applying  Al  techniques  to  individual 
applications  programs. 


Linking  Embedded  Performance  Measurement  with  Embedded  Training 


Technology  Development  Area:  The  Air  Force  has  no  way  of  determining  the  effectiveness 
of  embedded  training  in  general,  or  of  specific  embedded  training  features  in  relation  to  certain 
jobs.  AFHRL/OT  could  concentrate  on  developing  performance  measures  for  one  or  two  critical 
job  positions  (e.g.,  pilot,  co-pilot,  navigator,  electronic  warfare  officer).  A  task  analysis  of  the 
selected  position(s)  (describing  both  job  tasks  and  required  skill  levels)  would  need  to  be  made 
available,  either  from  existing  documentation  or  by  conducting  one.  Based  on  the  task  analysis 
data,  AFHRL/OT  would  first  need  to  determine  what  types  of  performance  data  are  actually 
necessary  to  collect.  Then  it  would  be  necessary  to  determine  the  feasibility  of  embedding 
the  desired  features  for  adequate  performance  measurement  in  the  host  equipment  in  terms 
of  costs  and  training  effectiveness  trade-offs. 

In  a  similar  manner,  AFHRL/ID  could  determine  what  performance  measures  are  required 
for  general  purpose  computer  systems,  then  include  the  appropriate  performance  measures  in 
embedded  Al  job  aids  for  applications  programs. 

Potential  Customer:  The  level  of  performance  measurement  in  most  embedded  training 
systems  currently  in  use  varies  considerably.  ASD/ENE  (Directorate  of  Support  Systems 
Engineering)  has  requested  that  research  be  undertaken  in  the  area  of  embedded  training 
performance  assessment.  Similar  needs  exist  at  the  SSC  to  link  on-the-job  performance  with 
on-the-job  training  in  general  computer  software  applications.  Our  study  has  indicated  that 
there  is  little  "hard'1  evidence  (i.e,,  performance  measurement  data)  for  claims  of  improved 
operational  readiness  due  to  embedded  training.  Research  into  performance  measurement 
requirements  for  embedded  training  can  provide  both  the  information  required  to  design  more 
effective  systems,  and  to  assess  force  readiness  at  a  very  discrete  level.  Performance 
measurement  data  can  be  useful  at  the  lowest  echelons  in  the  operational  command  in 
identifying  operators  who  require  additional  training  or  practice,  and  exactly  what  areas  these 
individuals  need  practice  in.  At  MAJCOM  levels,  performance  measurement  data  can  be 
accumulated  to  pinpoint  weaker  units  for  more  training  emphasis  or  other  assistance,  to  identify 
stronger  units  for  rewards,  and  to  assess  the  overall  ability  of  the  command  to  carry  out  its 
mission. 

Relevant  AFHRL  Strengths:  AFHRL/ID  has  developed  performance  measures  for  training 
effectiveness  evaluation;  AFHRL/LR  has  focused  its  efforts  specifically  on  the  evaluation  of  C2 
skills;  and,  AFHRL/OT  has  concentrated  on  performance  measurement  for  aircrew  training. 
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Risks:  Performance  measurement  in  anything  other  than  a  training  environment  is  traditionally 
looked  upon  as  a  threat.  This  is  particularly  true  when  the  performance  measurement  system 
judges  how  well  someone  does  his  job.  Although  performance  measurement  is  frequently 
considered  to  be  a  “good"  feature  of  embedded  training  systems,  it  still  must  gather  the  right 
kinds  of  information,  assess  it,  and  make  proper  use  of  it.  Major  command  support  may  be 
difficult  to  obtain  unless  increased  operational  readiness  can  be  shown  as  a  result. 


Embedded  T raining  within  the  Total  Training  System 


Technology  Development  Area :  Considerations  of  embedded  training  usually  are  not 
incorporated  into  total  training  systems  planning.  The  occurrence  of  embedded  training  in 
systems  seems  to  have  been  the  result  of  engineers  trying  to  incorporate  a  means  for  operators 
to  practice  using  the  system.  If  one  starts  to  think  about  embedded  training  during  systems 
development,  one  is  too  late  to  affect  the  configurations  of  either  the  system  or  the  training 
for  the  system.  The  selection  of  embedded  training  as  a  training  medium  and  the  determination 
of  what  embedded  training  features  to  include  should  be  a  part  of  the  system  concept.  They 
should  be  included  in  training  planning  from  the  very  first  steps  of  the  system  specification 
and  design  process. 

It  is  clear  that  embedded  training  should  be  considered  within  the  context  of  a  total  training 
system  approach.  The  question  is:  Where  does  it  fit  in?  AFHRL/LR's  contacts  at  AFSPACECOM 
would  be  needed  to  set  up  a  project  which  assessed  the  entire  spectrum  of  Space  Training, 
including  embedded  training.  The  necessary  research  would  probably  involve  an  analysis  of 
the  media  characteristics  of  embedded  training  (i.e.,  its  various  features).  Some  of  these 
features  could  be  incorporated  into  an  existing  system  and  their  effect  on  the  training  system 
assessed.  This  type  of  project  would  probably  best  be  undertaken  after  some  earlier  "groundwork" 
research  on  embedded  training  had  been  done,  and  there  was  more  widespread  recognition 
of  the  roles  of  embedded  training.  This  work  could  probably  draw  from  and  build  on  ARI’s 
project  on  "Embedded  Training  Utilization  Tools"  (scheduled  for  completion  in  Fiscal  Year  1992). 

The  discussion  above  concentrates  on  existing  embedded  training  systems.  A  similar  project 
could  also  be  undertaken  with  a  new  developing  system.  The  more  likely  customers  in  that 
case  would  be  ASD  for  airborne  systems  and  ESD  for  electronic  systems.  In  either  event, 
the  impact  of  embedding  training  into  a  new  system  versus  into  an  existing  system  might 
provide  an  interesting  contrast  for  research. 

Potential  Customer:  AFSPACECOM  and  ESD  have  a  number  of  existing  fielded  systems 
which  could  benefit  from  embedded  training  research.  BMEWS,  PAVE  PAWS  and  other  similar 
systems  have  been  using  embedded  training  since  their  fieldings;  however,  the  impact  of  these 
embedded  training  systems  on  what  needs  to  be  taught  in  formal  classroom  training  or  on  the 
total  training  system  has  not  been  examined.  AFSPACECOM  has  had  a  persistent  problem 
when  it  comes  to  the  total  training  picture.  AFSPACECOM  has  numerous  different  kinds  of 
systems  at  multiple  sites  worldwide  for  which  skilled  operators  must  be  prepared.  "Generic" 
training  at  a  central  facility  does  not  always  provide  the  needed  training  for  operators.  Without 
some  kind  of  embedded  training  for  these  systems,  AFSPACECOM  has  no  effective  way  of 
dealing  with  this  problem.  In  addition,  ESD  program  managers,  particularly  newly  assigned 
ones  without  the  benefit  of  experience  in  developing  systems,  have  no  basis  for  selecting  the 
proper  embedded  training  features.  Assistance  is  needed  for  them  in  assessing  the  impacts 
of  embedded  training  on  the  acquisition  and  development  of  systems. 

Relevant  AFHRL  Strengths:  AFHRL/MO’s  MPT  program  is  particularly  relevant  in  assessing 
total  training  requirements.  The  concept  behind  AFHRULR’s  Concurrent  Engineering  work  is 
also  applicable  to  a  project  for  determining  total  training  requirements  for  a  system.  AFHRL/LR's 
contacts  with  AFSPACECOM  would  be  beneficial  in  setting  up  any  program  with  them. 
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Risks:  There  is  a  risk  that  without  the  proper  kinds  of  guidelines,  embedded  training  will 
be  selected  for  training  when  it  is  inappropriate  or  that  it  may  not  be  selected  for  training 
when  it  is  the  most  appropriate  medium.  Even  more  likely  is  the  problem  of  identifying  the 
required  embedded  training  features  for  a  system  early  enough  in  the  acquisition  and  development 
process. 


Determining  Costs  of  Embedded  Training 


Technology  Development  Area:  AFHRL  could  gather  cost  data  from  a  wide  variety  of 
existing  embedded  training  systems  (the  systems  identified  in  Appendix  A  of  this  report  could 
be  a  start).  It  would  probably  be  necessary  to  work  closely  with  the  system  program  managers 
(perhaps  through  ESD)  to  sort  out  the  specific  costs  of  the  embedded  training  component, 
including  the  exact  sources  of  the  costs.  We  would  suppose  that  system  stimulation  might 
be  a  low-cost  item  for  most  signal  processing  systems.  Some  embedded  training  features 
which  may  contribute  to  increased  cost  may  include  making  scenario  production  user-friendly 
for  supervisors;  the  transportability  of  scenarios  from  one  system/site  to  another;  mixing  simulated 
and  live  signals  (or  the  suppression  of  one  or  the  other);  and  the  capability  to  conduct  multi-site 
training. 

This  issue  goes  hand-in-hand  with  determining  training  effectiveness.  The  costs  of  adding 
an  embedded  training  component  (or  specific  embedded  training  features)  to  a  computerized 
system  must  be  one  of  the  considerations  in  determining  the  appropriate  media. 

Ideally,  this  project  would  result  in  some  kind  of  automated  tool  which  would  allow  training 
planners  or  program  managers  to  assess  the  cost  and  safety  trade-offs  that  might  result  from 
their  decision  to  add  a  particular  embedded  training  capability  to  a  system.  AFHRL  could 
probably  build  upon  the  work  conducted  by  ASD/XR  on  Automated  Training  Projection 
Methodologies  (O’Brien  &  Hess,  1988c).  However,  unlike  the  ASD/XR  effort,  AFHRL’s  work 
should  be  based  upon  and  incorporate  a  significant  amount  of  foundational  research  on 
embedded  training. 

Potential  Customer:  Our  study  showed  that  most  people  are  unaware  to  what  extent 
embedded  training  adds  to  the  cost  of  a  computerized  system.  These  individuals  ate  also 
unaware  of  the  various  types  of  embedded  training  features  which  are  available  for  them  to 
use  in  designing  the  specific  embedded  training  component  of  their  system,  and  of  the  degree 
of  difficulty  associated  with  developing  each.  ESD  program  managers  are  not  accustomed  to 
dealing  with  training,  let  alone  how  various  embedded  training  features  can  be  used  to  address 
the  specific  training  requirements  of  a  large  computerized  system.  Therefore,  the  program 
managers  are  not  in  the  habit  of  estimating  the  impact  of  various  embedded  training  features 
either  on  the  technology  of  the  system  being  developed  or  on  its  costs.  The  determination 
of  some  basic  embedded  training  cost  factors  and  a  decision  aid  to  help  program  managers 
are  needed  by  ESD. 

Relevant  AFHRL  Strengths:  AFHRL/ID’s  work  on  the  Training  Decisions  System  (TDS)  has 
shown  that  it  is  clearly  capable  of  dealing  with  training  cost  data.  AFHRL  appears  to  have 
a  strong  commitment  to  providing  Air  Force  customers  with  decision  aiding  tools,  as  evidenced 
by  AFHRL/ID's  involvement  in  TDS,  and  AFHRL/LR’s  involvement  in  CAD/CAE  design  tools.  A 
similar  decision  aid  for  embedded  training  would  be  very  consistent  with  these  activities. 

Risks:  The  risks  of  such  a  study  could  be  minimized  if  it  were  performed  in  conjunction 
with  other  embedded  training  research  efforts.  The  only  obstacle  is  that  in  order  for  the  study 
to  be  performed,  AFHRL  will  have  to  work  with  cost  data  which  have  traditionally  not  been 
itemized  in  budgets  for  systems.  AFHRL  would  need  the  system  program  managers’  cooperation 
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plus  some  technical  expertise  to  sort  out  which  budget  items  relate  to  the  embedded  training 
component,  and  to  what  extent  various  embedded  training  features  add  to  the  cost  of  the 
system. 


Model  Specifications  for  Embedded  Training 


Technology  Development  Area:  Embedded  training  systems  tend  to  vary  widely,  and  the 
features  found  from  one  embedded  training  system  to  the  next  are  not  standardized.  Guidelines 
for  the  development  of  embedded  training  could  contribute  to  the  standardization  of  embedded 
training  features  across  similar  systems. 

This  R&D  should  be  the  culmination  of  earlier  research  on  embedded  training.  To  be 
maximally  useful,  the  study  should  probably  result  in  some  type  of  automated  decision  id 
which  would  (a)  allow  training  planners  to  see  the  effect  of  decisions  they  make  with  regard 
to  embedded  training,  in  terms  of  cost  and  effect  on  the  host  equipment;  and  (b)  produce 
some  kind  of  "boilerplate"  which  could  be  tailored  for  embedded  training  specifications  based 
on  inputs  to  the  decision  aid.  The  lead  AFHRL  division  for  this  R&D  effort  could  be  either 
AFHRL/LR  or  AFHRL/OT,  depending  on  whether  the  customer  was  ESD  (concerned  with 
electronic  systems)  or  ASD  (concerned  with  aircraft  systems). 

Potential  Customer:  Both  ESD  and  ASD  need  assistance  in  properly  defining  the  required 
embedded  training  features  for  systems  they  are  acquiring  or  developing.  Program  managers 
in  these  organizations  are  unaware  of  embedded  training  and  its  impact  on  computerized 
systems. 

Relevant  AFHRL  Strengths :  Much  of  AFHRL’s  work  has  been  oriented  toward  helping  the 
Air  Force  be  a  "smart  shopper"  through  the  development  of  "model"  specifications  for  systems. 
AFHRL  is  certainly  familiar  with  specifications  development.  Some  training  systems  for  which 
AFHRL  has  been  involved  in  specifications  development  include  the  C-130  Model  Aircrew 
Training  System  and  the  B-52/KC-135  Combat  Crew  Training  System  (CCTS)  Modernization 
(AFHRL/OT),  and  Command  and  Control  Team  Training  (AFHRL/LR).  Although  none  of  these 
are  embedded  training  systems,  they  still  demonstrate  AFHRL’s  overall  sensitivity  to  training 
needs  and  issues  through  the  development  of  model  specifications.  AFHRL/ID’s  experience  in 
developing  decision  aids  for  training  planners  should  also  be  very  useful. 

Risks:  There  is  a  risk  that  the  computerized  weapon  systems  selected  to  model  may  be 
designed  with  embedded  training  in  mind,  that  the  embedded  training  component  of  the  system 
will  become  a  driving  factor  in  system  design  and  development  rather  than  be  a  part  of  the 
total  training  package  for  the  weapon  system.  Embedded  training  should  never  be  forced  on 
a  system.  The  possibility  for  failure  is  high  if  the  embedded  training  does  not  properly  "fit" 
the  weapon  system.  For  example,  the  Training  Interface  Unit  (TIU),  an  embedded  training 
component  of  the  Army’s  "Aquila”  Remotely  Piloted  Vehicle,  was  intended  to  train  system 
operators  in  flying  skills,  payload  operations,  and  engaging  targets;  however,  according  to  Army 
sources,  the  TIU  had  only  limited  usefulness  because  its  portrayal  of  targets  and  artillery  bursts 
lacked  realism. 


Potential  R&D  Agenda 


As  a  result  of  having  identified  many  of  the  issues  facing  embedded  training  designers, 
developers,  and  users  during  this  study,  we  have  suggested  some  potential  R&D  technology 
development  areas  that  AFHRL  can  explore  in  meeting  the  Air  Force  s  needs  for  embedded 
training.  Now,  we  will  outline  ways  for  AFHRL  to  accomplish  this  task  (i.e.,  an  embedded 
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training  R&D  strategy  for  AFHRL).  The  proposed  embedded  training  R&D  agenda  is  summarized 
in  Table  2  and  provides  a  basis  for  identifying  AFHRL’s  role. 


Table  2.  AFHRL  Potential  R&D  Agenda 


Technology  area 

Potential  project 

Potential  customers 

1.  Safety  of  ET 

Characterization  of 

Existing  ET  Features 

AFSPACECOM 

Airborne  ET 

ASD 

2.  Intelligent 

ET  for  Financial 

A&F  Center 

Tutors  for 

Applications 

Standard  Systems 

Al  Aids  for  Application 
Software  Training 

SSC 

3.  Embedded  Performance 

Airborne  ET 

ASD 

Measurement  and  ET 

ET  in  Standard  Systems 

SSC 

4.  ET  in  Total 

Determination  of 

AFSPACECOM 

Training  Systems 

ET  Features 

5.  ET  Costs 

Cost  Effectiveness  of 

ET  Features 

ESD 

ET  Decision  Aid 

ESD 

6.  Model  ET 

Model  Specs  for 

ASD 

Specifications 

Airborne  Systems 

Model  Specs  for 

Radar  Systems 

ESD,  AFSPACECOM 

ET  Decision  Aid 

ASD,  ESD 

Implementation  Strategies 


Joint  Service  Activities 


If  AFHRL  decides  to  become  more  involved  in  embedded  training  R&D,  we  would  recommend 
that  it  also  become  actively  involved  in  the  JTCG-TSD  Embedded  Training  Working  Group.  To 
our  understanding,  AFHRL/OT  is  AFHRL's  representative  on  the  JTCG-TSD.  However,  the 
degree  of  participation  by  that  division  has  been  limited. 

We  would  also  recommend  that  the  Air  Force  follow  its  own  R&D  agenda  rather  than  simply 
adopting  what  has  been  proposed  by  the  JTCG-TSD.  There  are  two  reasons  for  this.  First, 
the  Air  Force  has  some  distinctly  different  needs,  weapon  systems,  training  requirements,  and 
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procurement  policies  from  those  of  the  other  Services.  Therefore  JTCG-TSD  products  like 
ARI’s  10-volume  set  of  Guidelines  for  Implementing  Embedded  Training  will  need  further 
modification  before  they  can  fully  address  Air  Force  concerns,  or  before  they  can  be  used  by 
Air  Force  customers.  The  Air  Force  needs  research  products  which  directly  address  Air  Force 
needs. 

Second,  and  perhaps  more  importantly,  the  other  Services  are  pursuing  embedded  training 
research  projects  even  as  they  increase  their  use  of  embedded  training.  Both  the  Army  and 
Navy  already  promote  and  widely  use  embedded  training.  The  other  Services  already  have 
in  place  policies  concerning  the  consideration  of  embedded  training  as  a  medium  and  policies 
for  its  use.  Both  the  Army  and  Navy  apparently  have  no  problem  in  selecting  embedded 
training  as  a  medium  or  in  developing  embedded  training  systems.  In  contrast,  Air  Force 
users  and  potential  users  of  embedded  training  appear  to  be  more  cautious,  and  will  probably 
require  more  research  in  support  of  embedded  training  before  they  are  willing  to  endorse  its 
use.  We  believe  that  the  Air  Force  needs  to  pursue  a  step-by-step  strategy  of  conducting 
embedded  training  research  which  eventually  culminates  in  the  development  of  guidelines  for 
the  acquisition  and  development  of  embedded  training  prior  to  its  extensive  use  throughout  the 
force. 


Projects,  Customers,  and  Resources 


Three  AFHRL  Divisions  have  a  high  potential  for  effectively  engaging  in  embedded  training 
R&D:  the  Operations  Training  Division  (AFHRL/OT),  the  Training  Systems  Division  (AFHRL/ID), 
and  the  Logistics  and  Human  Factors  Division  (AFHRL/LR).  Each  of  these  Divisions  has 
unique  capacities  which  allow  it  to  perform  specific  embedded  training  R&D.  These  capacities 
and  the  potential  for  embedded  training  R&D  associated  with  each  are  summarized  below. 

The  Operations  Training  Division  (AFHRL/OT): 

Projects:  The  potential  for  embedded  training  aboard  aircraft  remains  high  on  the  list  of 
R&D  issues  to  be  explored.  However,  there  will  be  relatively  little  payoff  in  the  near  term  for 
projects  supporting  ASD’s  embedded  training  needs.  ASD’s  concerns  focus  on  the  safety  of 
embedded  training.  Of  all  the  Laboratory  divisions,  we  feel  that  AFHRL/OT  can  best  support 
A3D  in  finding  a  solution  to  its  embedded  training  R&D  problems. 

Customer:  ASD/ENE  has  already  shown  an  interest  in  OT’s  assisting  with  embedded 

training  R&D.  If  AFHRL  is  interested  in  supporting  the  flying  MAJCOMs  through  embedded 

training  R&D,  it  will  have  to  market  other  segments  within  ASD  (namely  the  SPOs),  and  gain 
the  interest  of  SAC,  TAC,  and  MAC. 

Equipment:  Some  use  could  probably  be  made  of  AFHRL70T's  existing  simulator  capability. 
In  addition,  ASD/ENE  could  probably  identify  a  candidate  aircraft  (or  avionics  subsystem)  to 
serve  as  a  testbed  for  embedded  training  research  (e.g.,  the  ATF  program). 

Personnel:  AFHRL/OT  has  an  appropriate  mix  of  professional  staff  and  contractors  to 

support  aircraft  embedded  training  R&D.  Technical  skills  in  aircrew  training  and  complementary 
engineering  skills  provide  the  right  combination  for  embedded  training  R&D  to  be  successful. 


The  Training  Systems  Division  (AFHRL/ID): 

Projects:  Based  on  AFHRUID’s  past  track  record,  the  Al  projects  proposed  in  this  report 
are  highly  likely  to  succeed  if  undertaken  by  that  division.  The  definite  but  unrecognized  need 
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for  embedded  training  assistance  by  the  SSC  could  produce  several  projects  if  marketed 
properly.  AFHRL/ID  must  work  hard  to  gain  support  from  SSC  for  any  embedded  training 
research  task.  However,  once  the  SSC  has  seen  the  benefits  to  be  derived  from  embedded 
training,  they  should  become  a  much  more  willing  customer.  AFHRL/ID  could  also  build  on 
its  research  in  Job  Performance  Measurement  (JPM).  AFHRLVID’s  JPM  and  Al  capabilities 
combine  to  provide  the  Laboratory  with  a  unique  opportunity  to  assist  SSC  with  some  of  its 
more  difficult  software  training  problems. 

Customer:  The  critical  problem  facing  AFHRL/ID  is  to  market  a  customer  for  its  potential 
embedded  training  R&D  products.  As  we  see  it,  the  most  likely  potential  customer  for  AFHRL/ID 
is  the  SSC.  However,  SSC  has  not  recognized  embedded  training  as  a  potential  solution  to 
their  problems  and  will  need  to  be  sold  on  it.  When  approaching  SSC,  ID  might  well  focus 
on  solving  SSC’s  training  problems  rather  than  on  "selling"  embedded  training. 

Equipment:  Al  development  platforms  and  the  appropriate  software  tools  will  be  needed 
for  any  embedded  training  project  using  Al  in  support  of  the  SSC.  The  source  codes  for  one 
or  more  standard  systems,  programmers  who  understand  them,  and  the  appropriate  computer 
hardware  will  also  be  required. 

Personnel:  Marketing  skills  in  the  person  selected  to  head  up  this  embedded  training  effort 
will  be  essential  to  its  success.  No  projects  have  been  proposed  by  the  potential  customer. 
SSC  currently  feels  that  they  are  able  to  do  it  themselves.  Anyone  approaching  SSC  must 
be  able  to  convince  them  that  they  need  the  research. 

The  Logistics  and  Human  Factors  Division  (AFHRL/LR): 

Projects:  Several  AFSPACECOM  projects  were  suggested  which  take  advantage  of  the 

existing  skills  in  this  division.  Most  of  these  projects  will  likely  have  a  relatively  shorter-term 
duration  (i.e.,  2  to  3  years)  and  only  a  moderate  risk  factor.  We  fee)  that  AFHRL/LR  can 
best  support  AFSPACECOM’s  training  needs.  AFHRL/LR  can  also  provide  needed  support  to 
ESD  program  managers  in  recognizing  requirements  for  embedded  training  and  being  able  to 
cost-effectively  select  the  proper  embedded  training  features  necessary  for  a  system. 

Customer:  AFSPACECOM  has  already  expressed  an  interest  in  working  with  AFHRL.  The 
critical  factor  in  engaging  in  embedded  training  R&D  with  AFSPACECOM  is  to  approach  them 
with  the  intention  of  improving  their  overall  training  and  readiness  posture.  ESD  must  be 
marketed  as  a  potential  customer.  There  is  a  greater  likelihood  of  success  at  ESD  because 
of  the  direct  connection  with  AFSPACECOM  systems. 

Equipment:  Any  work  done  will  have  to  be  compatible  with  existing  AFSPACECOM  systems. 
This  may  entail  having  a  similar  computer  architecture  set  up  as  a  testbed  for  AFHRL/LR  or 
being  associated  with  the  contractor  developing  an  AFSPACECOM  system. 

Personnel:  AFHRL/LRG  (the  Laboratory  branch  with  the  strongest  contacts  at  AFSPACECOM) 
has  its  current  staff  resources  spread  across  several  projects.  This  could  pose  a  serious 
problem  in  taking  on  any  additional  work  in  support  of  embedded  training  R&D. 


Commitment  and  Leadership 


In  order  to  undertake  embedded  training  research,  the  management  of  AFHRL  must  make 
a  positive  commitment  to  doing  so.  In  each  of  the  cases  described  in  this  study,  we  have 
pointed  out  the  strong  necessity  for  the  division  involved  to  be  able  to  market  their  capabilities 
to  potential  customer(s).  The  Laboratory  must  also  commit  to  designating  and  supporting 
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someone  within  AFHRL  as  the  focal  point  for  embedded  training  R&D.  This  individual  does 
not  have  to  be  proficient  in  embedded  training  design  or  development,  but  he  or  she  should 
be  well  respected  technically  and  have  an  insight  into  training  problems  and  their  potential 
solutions.  The  Laboratory’s  focal  point  for  embedded  training  should  also  be  appointed  to  the 
Joint  Technical  Coordinating  Group’s  Embedded  Training  Working  Group. 

In  our  opinion,  one  of  the  keys  to  the  success  of  the  AFHRL  embedded  training  R&D 
agenda  is  performing  some  embedded  training  R&D  task  as  soon  as  possible  to  solidify  the 
Laboratory’s  credibility  in  the  field.  The  Laboratory's  Commander  may  need  to  exert  personal 
influence  to  market  potential  customers  and  inaugurate  an  initial  embedded  training  task. 

Our  initial  concept  was  that  a  single  AFHRL  division  should  take  the  lead  in  embedded 
training  R&D  to  the  exclusion  of  the  other  divisions.  We  have  modified  our  position  somewhat 
based  on  the  potential  customers  and  the  relationships  already  established  with  them  by  some 
of  the  divisions.  We  now  feel  that  AFHRL/LR  appears  to  have  the  greatest  probability  of 
being  able  to  obtain  and  sustain  support  for  its  embedded  training  R&D  tasks.  However,  we 
also  feel  that  each  division  should  pursue  one  or  two  embedded  training  customers.  AFHRL/OT 
should  concentrate  initially  on  ASD  and,  then,  on  the  CONUS  flying  commands.  AFHRL/ID 
should  try  to  capitalize  on  its  decision  aiding  and  Al  expertise  by  focusing  on  the  SSC.  In 
general,  initial  research  should  focus  on  the  issues  described  in  this  study,  then  branch  out 
as  the  customer  becomes  more  accustomed  to  dealing  with  embedded  training. 
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APPENDIX  A:  EMBEDDED  TRAINING  SYSTEMS 


Air  Force  Embedded  Training  Systems 

AMHS  -  Automated  Message  Handling  System  [proposed] 

ATF  -  Advanced  Tactical  Fighter  [proposed] 

AWACS  -  Airborne  Warning  and  Control  System 
BCAS  -  Base  Contracting  Automated  System 
BMEWS  -  Ballistic  Missile  Early  Warning  System 
CAMS  -  Core  Automated  Maintenance  System  [proposed] 

CAS  -  Combat  Ammunition  System 

CBTS  -  Computerized  Base-level  Training  System 

CMOS  -  Cargo  Movement  System  [proposed] 

COOF  -  Comptroller  Office  of  the  Future  Embedded  Training  [proposed] 

Embedded  Training  for  Tactical  Aircraft  Project 

F-16  OBEWS  -  F-16  On-Board  Electronic  Warfare  System 

GPMCS  -  Global  Positioning  System  Master  Control  Station 

IBM  4381,  CPCI-235  module 

MPRS  -  Medical  Personnel  Records  System 

OBS  F-15  -  On-Board  Simulation  System 

OTH-B  -  Over-the-Horizon  Backscatter  Radar  [proposed] 

PACCITS  -  PACAF  Command  &  Control  Info.  &  Training  Sys.  [proposed] 
PAVE  PAWS  -  PAVE  Phased  Array  Warning  System 
SRS  -  Satellite  Readout  Station 
TEMPLAR  Embedded  Training  [proposed] 

WWMCCS  -  Worldwide  Military  Command  and  Control  System 
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Army  Embedded  Training  Systems 


AFATDS  -  Advanced  Tactical  Data  System 

AN/TPQ-29  -  Hawk  Missile  System  Radar  Signal  Stimulator 

AN/TSQ-73  -  Missile  Minder  Command  and  Control  System 

FAAD  NLOS  -  Forward  Area  Air  Defense  Non  Line  of  Sight 

FOQ-M  -  Fiber  Optic  Guided  Missile 

HIP  -  Howitzer  Improvement  Program 

MCS  2  -  Maneuver  Control  System  2 

MLRS  -  Multiple  Launch  Rocket  System 

Patriot  Missile  System  Embedded  Training 

TACFIRE  -  Tactical  Fire  Direction  System 

Till  -  Training  Interface  Unit 
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Navy  Embedded  Training  Systems 


ACTS  -  Aegis  Combat  Training  System 
ADSIM  -  Automatic  Detection  Tracking  System  Simulation 
ADTSM  -  Automatic  Detection  and  Tracking  Simulator 
AN/BSY-2  -  Combat  Control  Onboard  Training 

AN/SQQ-89  -  Surface  Anti-Submarie  Warfare  (ASW)  Combat  System 
AN/SQR-OBT 

AN/SQS-56  Target  Simulator 
AN/SQS-T5 

CATCC  STSS  -  Carrier  Air  Control  Ctr.  Shipboard  Simulation  Sys 
CSTS  -  Combat  Simulation  Test  System 
CSTS  -  Combat  System  Test  Set 

CTORP  -  Combat  Team  Operational  Readiness  Program 

DPT  -  Deployable  Proficiency  Trainer 

ECM  -  Electronic  Counter  Measures  Generator 

GMTR  -  Guided  Missile  Training  Round 

Guided  Missile  Simulator,  MK  89  MODO 

IFT  -  In  Flight  Training  System 

L-TRAN  -  Lesson  Translator 

LHD-1  CSTS  -  Combat  Simulation  Test  System,  AN/SSQ-91 

ORATS  -  Operational  Readiness  Assessment  and  Training  System 

OSMOS  -  Own  Ship  Motion  Simulator 

OTS  -  Operational  Training  Software  for  SLQ-32 

PME  -  Performance  Measurement  and  Evaluation  for  Sonar 

PME  -  Performance  Measuring  Equipment  for  SQQ-23 

RACOR  -  Radar  Recorder 
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Navy  Embedded  Training  Systems  (Concluded) 


Radar  Proficiency  Simulator 

RAVIR/MCST  -  Radar  Video  Recorder/T-3  Mobile  Combat  Systems  Trainer 

REES-204  -  Radar  Electromagnetic  Environmental  Simulator 

RESS  -  Radar  Environmental  Simulator  System,  AN/USQ-93 

RFTT  -  Radio  Frequency  Target  Generator 

RVS  -  Radar  Video  Simulator 

SEAT  -  System  Evaluator,  Trainer,  MK  16  MOD  1 

SETS  -  SPA  -25G  Embedded  Training  System 

Silverbox  2/WLR-1 

SORATS  -  Submarine  Operational  Readiness  Assessment  and  Training  System 
STSS  -  Sonar  Target  Signal  Simulator 
STTP  -  Simulated  Target  Training  Program 
Tactical  Modular  Display 

TM  CCS  -  Training  Mode  of  Combat  Control  System 

TTP  -  Tactical  Proficiency  Program 

TSAM  -  Training  Surface  to  Air  Missile 

Unit  34,  Sonar  Target  Generator 

VSSG  VSS-SM  31 9A  -  Video  Signals  Simulator 
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APPENDIX  B:  EMBEDDED  TRAINING  TELEPHONE  QUESTIONNAIRE 


Instructions: 

Please  provide  as  much  of  the  following  information  as  possible  regarding 
the  designated  Embedded  Training  (ET)  system.  If  you  are  unable  to 
answer  the  question  or  to  provide  any  data,  indicate  such  and  provide  a 
reason  (e.g.,  Not  Available,  Classified,  etc.).  If  you  have  more  data  than 
can  easily  be  transcribed  on  the  form,  please  attach  whatever  documentation 
you  feel  is  pertinent.  Feel  free  to  indicate  the  name(s)  and  contact  phone 
(commercial  and  AUTOVON)  or  personnel  who  may  be  a  source  of  additional 
details: 


Section  I  -  System  Descriptive  Information 

1.  ET  system  name: 

2.  ET  system  developer: 

3.  Host  system  name  (if  different  from  ET  System): 

4.  Host  system  developer: 

5.  Platform: 

6.  Sponsor  Organization: 

7  ET  added  before/after  host  system  became  operational? 

8.  Does  the  ET  require  the  use  of  additional  special  equipment?  If  so, 
please  describe.  _ 


9.  On  a  scale  of  1  to  5,  indicate  the  degree  to  which  each  of  the  following 
factors  influenced  the  selection  of  ET  as  a  training  medium  (1  =  little  or 
no  influence;  5  =  great  influence;  DK  =  don’t  know). 


a. 

Internal  policy  requirements 

1 

2 

3 

4 

5 

DK 

b. 

Avai'ability  of  operator  slack  time 

1 

2 

3 

4 

5 

DK 

c. 

Dispersed  location  of  trainees 

1 

2 

3 

4 

5 

DK 

d. 

Cost  benefits 

1 

2 

3 

4 

5 

DK 

e. 

Security 

1 

2 

3 

4 

5 

DK 

f. 

Need  to  make  training  available  to 
reservists  or  other  "part-time1' 
personnel 

1 

2 

3 

4 

5 

DK 

g 

Need  for  operator  refresher  training 

1 

2 

3 

4 

5 

DK 

h 

Media  selection  process 

1 

2 

3 

4 

5 

DK 

' 

Other 

1 

2 

3 

4 

5 

DK 
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10.  On  a  scale  of  1  to  5,  indicate  to  what  extent  each  of  the  following  was 
used  in  development  of  the  embedded  training  (1  =  not  used;  5  =  much 
used;  DK  =  don't  know). 


a. 

Job/Task  Analysis 

1 

2 

3 

4 

5 

DK 

b. 

Training  Objectives 

1 

2 

3 

4 

5 

DK 

c. 

Host  System  documentation 

1 

2 

3 

4 

5 

DK 

d. 

Host  System  developers’  advice 

1 

2 

3 

4 

5 

DK 

e. 

Other 

1 

2 

3 

4 

5 

DK 

If  any  of  the  above  were  used,  please  attach  a  copy,  if  available. 

Section  II  -  ET  System  Training  Information 

11.  What  skills  is  the  system  intended  to  teach?  _ 


12.  What  kind  of  training  goes  on?  (check  all  that  apply) 


a. 

1 

]  Basic/Entry 

[ 

]  Continuation  [ 

]  Upgrade 

b. 

[ 

]  Individual 

[ 

j  Team  [ 

j  Sub-team 

c. 

[ 

]  Operator 

[ 

]  Maintenance 

d. 

[ 

]  Complete  scenario 

[ 

]  Specific  Function(s) 

13.  Please  indicate  specialty  of  personnel  being  trained  on  this  system.  Use 
specialty  codes  (i.e.,  AFSC,  MOS,  l.tC)  if  possible. 


14.  Is  user  documentation  available?  [  J  Yes  [  ]  No  [  j  DK 
If  yes,  which  of  the  following  does  it  include? 

Please  check  all  that  apply. 

[  ]  Installation  and  set-up  instructions 
[  ]  Instructional  guidance 
[  ]  Training  objectives 
[  ]  Help  screens 
[  j  Other  (please  describe): 
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15.  Are  on-line  scenarios  available?  [  ]  Yes  [  ]  No  [  ]  DK 
If  yes,  which  of  the  following  features  are  available? 

Please  check  all  that  apply. 

[  ]  Scenario  authoring  capability 
[  ]  Canned  scenarios 
[  ]  Pause  capability 
[  ]  Fast  forward  capability 
[  ]  Other  (please  describe): 


16.  Is  student  performance  recorded?  [  ]  Yes  [  ]  No  [  ]  DK 
If  yes,  which  of  the  following  features  are  available? 

Please  check  all  that  apply. 

[  ]  Playback  of  students'  sessions 
[  ]  Storage  of  student  performance  records 
[  ]  Other  (please  describe): 


17  Is  on-line  feedback  available?  (  ]  Yes  [  ]  No  [  ]  DK 
If  yes,  which  of  the  following  features  are  available? 

Please  check  all  that  apply. 

[  ]  Diagnostic  feedback 
[  ]  On-line  remediation 
[  ]  Other  (please  describe): 


18  Development  status  of  the  ET. 

Drawing  board  [  ]  Prototyped  [  j  In  testing  [  ]  Fielded  [  ] 

If  in  testing  or  already  fielded,  complete  Section  III;  otherwise  skip  to  Section 
IV. 
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Section  III  -  System  Usage  Information 
A.  Deployment  sites 


19.  Describe  the  types  of  sites  where  the  ET  system  has  been  deployed: 


20.  List  the  sites  where  the  ET  system  is  being  deployed  at  present: 


21. 

How  many  and  where  are  future  sites  for  deployment? 


B_.  Frequency  of  use 

22.  Please  estimate  how  many  people  are  using  this  ET  system  at  present 
(e.g.,  #  per  month/year,  etc  ):  _ 


23.  On  a  scale  of  1  to  5,  how  would  you  characterize  the  extent  of  actual 

use  of  this  system  in  the  field  (1  =  seldom  or  never  used;  5  =  very 
frequently  used)?  1  2  3  4  5 

If  the  answer  to  question  23  is  1,  2,  or  3,  then  complete  question  24; 

otherwise  skip  to  question  25. 

24.  On  a  scale  of  1  to  5,  indicate  the  degree  to  which  each  of  the  following 
factors  may  have  contributed  to  the  system’s  not  being  more  frequently 
used  (1  =  not  a  factor;  5  =  major  factor;  DK  =  don’t  know). 


a. 

System  is  inoperative  while  ET  is  in 
use 

1 

2 

3 

4 

5 

DK 

b. 

ET  updates  lag  behind  host  system 
updates 

1 

2 

3 

4 

5 

DK 

c. 

ET  documentation  is  insufficient/ 
unclear 

1 

2 

3 

4 

5 

DK 

d. 

ET  is  inconvenient  to  set  up  or  use. 

1 

2 

3 

4 

5 

DK 

e. 

ET  is  frequently  down  or  bug-prone 

1 

2 

3 

4 

5 

DK 

f. 

Other 

1 

2 

3 

4 

5 

DK 

25  Has  the  effectiveness  of  the  ET  been  assessed? 

[  ]  Yes  [  ]  No  [  ]  DK 
If  yes, 

a.  Who  performed  the  assessment? 

b.  Were  the  results  positive  or  negative? 

c.  Has  the  assessment  had  any  effect 
on  modifications  to  the  ET  system? 

Please  provide  a  point  of  contact  for  this  information  (use  pg.  6);  and  attach 
a  copy  of  the  data,  if  available. 


Section  IV  -  Other 

26.  If  you  wish,  you  may  use  the  space  below  to  make  other  comments 
about  this  ET  system. 


27.  How  could  the  ET  be  improved? 


Section  V  -  Recommended  Contacts 

Please  recommend  any  points  of  contact  who  you  think  would  be  helpful. 

Name: 

Phone: 

Organization/Company: 

Function  (e.g.,  developed  ET,  program  manager,  etc.): 


Name: 

Phone: 

Organization/Company: 

Function  (e.g.,  developed  ET,  program  manager,  etc.): 


Name: 

Phone: 

Organization/Company: 

Function  (e.g.,  developed  ET,  program  manager,  etc.): 


Name: 

Phone: 

Organization/Company: 

Function  (e.g.,  developed  ET,  program  manager,  etc.: 


The  responses  to  this  inquiry  will  be  used  in 
statistical  analyses  across  systems,  and  will 
not  be  released  other  th'"»  to  AFHRL/SA 
without  prior  permission  o.  .e  respondent. 
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APPENDIX  C:  INFORMATION  TO  BE  GATHERED  DURING  SITE  VISITS 


Overview 


The  purpose  of  the  site  visits  is  to  gather  information  about  particular  embedded  training 
(ET)  systems  which  will  aid  in  the  development  of  R&D  strategies  in  ET  for  AFHRL.  Information 
will  be  gathered  in  the  following  categories: 


•  General  Information 

•  Cost  of  ET 

•  Responsibility  for  ET 

•  Resistance  to  ET 

•  Evaluation  of  ET 

•  ET  Development  Milestones 

•  Skills  Taught  via  ET 

•  Instructional  Range  of  ET 

•  Operational  Features  of  ET 

•  ET  System  Architecture 

•  User  Manuals  for  ET 

•  ET  System  Maintenance 

•  Trainees’  Response  to  ET 

•  Training  Supervisors’  Response  to  ET 

•  Scheduling  Use  of  ET 


The  following  section  describes  in  detail  the  type  of  information  desired  in  each  category. 


Detailed  Description 


General  Information 


Information  desired:  Name  of  host  system;  name  of  ET  system  (if  different  from  host); 
platform;  interviewees'  names,  job  titles,  and  organizational  affiliation. 


Cost  of  ET 


Information  desired:  Cost  of  the  ET;  cost  calculation  methods  or  tools  used;  use  or  non-use 
of  cost-effectiveness  analyses  in  selection  of  ET  as  a  training  medium. 


Responsibility  for  ET 


Information  desired:  Names  of  contractors  and  in-house  organizations  responsible  for  the 
ET,  their  roles,  and  qualifications.  Were  roles  clearly  delineated?  How  was  role  confusion 
resolved?  How  was  responsibility  for  training  transferred  to  ATC  or  to  field  training  supervisors 
after  the  system  was  fielded? 
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Resistance  to  ET 


Information  desired:  Sources  of  resistance,  if  any,  to  the  use  of  ET  and  their  concerns: 
ways  these  concerns  were  resolved.  What  types  of  information  or  data  would  have  helped 
to  alleviate  these  concerns? 


Evaluation  of  ET 


Information  desired:  Methods  used  to  evaluate  the  instructional  effectiveness  of  the  ET; 
methods  for  data  collection  and  analysis;  evaluation  results;  evaluation  scheduling;  provisions 
for  making  changes  based  on  evaluation  results. 


ET  Development  Milestones 


Information  desired:  Description  of  what  was  presented  for  ET  at  each  milestone  of  the 
host  system  development  cycle.  To  what  extent  was  ET  an  item  for  discussion  at  Air  Force 
Systems  Acquisition  Review  Council  (AF  SARC)  meetings? 


Skills  Taught  via  ET 


Information  desired:  Skill  areas  taught  by  the  ET;  specific  procedures  or  factors  used  to 
select  skill  areas  to  be  taught  by  the  ET.  Were  task  analyses  utilized  in  making  these 
decisions?  Were  desired  skill  areas  prioritized? 


Instructional  Range  of  ET 


Information  desired:  Role  of  ET  in  the  total  training  pipeline  for  the  trainee.  Which  of  R. 
Gagne’s  nine  events  of  instruction  are  provided  by  the  ET  and  which  are  provided  for  by  other 
forms  of  training  (see  attached  list  of  R.  Gagne’s  nine  events  of  instruction)?  Is  it  necessary 
or  desirable  to  use  ET  for  all  instructional  events? 


Operational  Features  of  ET 


Information  desired:  Description  of  features  offered  by  the  ET,  such  as  rewind,  fast  forward, 
freeze,  scenario  authoring,  performance  recording,  and  feedback;  most  commonly  used  features; 
desired  features.  For  performance  recording,  exactly  which  aspects  of  student  performance  are 
recorded?  For  feedback,  is  the  feedback  tailored  to  student  responses  or  just  correct/incorrect? 


ET  System  Architecture 


Information  desired:  Techniques  used  to  add  ET  to  the  host  system;  extent  to  which  ET 
data  are  handled  differently  from  real  data;  extent  to  which  system  architecture  was  influenced 
by  training  requirements. 
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User  Manuals  for  ET 


Information  desired:  Description  of  manuals  developed  to  guide  ET  set-up,  operation,  or 
use  in  training.  How  is  this  documentation  made  available  to  supervisors  and  trainees  using 
the  system?  Which  manuals  are  most  commonly  used? 


ET  System  Maintenance 


Information  desired:  Procedures  for  managing  malfunctions  in  or  updates  to  the  ET  system. 
How  is  the  cost  of  maintenance  accounted  for? 


Trainees*  Response  to  ET 


Information  desired:  Description  of  trainees'  reactions  to  ET  and  problems  they  have 
experienced  with  the  system. 


Training  Supervisors’  Response  to  ET 


Information  desired:  Description  of  training  supervisors'  reactions  to  ET;  problems  experienced 
in  using  ET  for  training  (e.g.,  problems  with  system  unavailability  or  instructional  inadequacy). 
What  features  have  they  commented  on? 


Scheduling  Use  of  ET 


Information  desired:  Typical  scheduling  of  ET  use  on  a  daily  basis, 
considered  to  be  training  or  time  on  the  job? 


Is  use  of  ET 


Summary 


The  interviewer  may  otmcturo  the  site  visit  in  any  appropriate  manner,  but  should  focus 
on  gathering  information  in  the  15  categories  descrioed  above  for  each  ET  system  being 
studied.  In  some  cases,  obtaining  the  desired  information  may  require  interviewing  representatives 
from  different  offices.  The  interviewer  should  keep  a  record  of  the  source  of  each  item  of 
information  obtained. 
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Provides  a  background  of  various  considerations  for  implementing  ET  into  systems,  and  for 
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set  of  guidelines  for  the  design  of  future  ET  capabilities.  Also  consolidates  lessons  learned 
with  regard  to  the  systems  designs,  capabilities,  and  their  effectiveness  for  training. 
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simulation.  Includes  the  requirement  for  analysis,  payoff,  and  measuring  payoff,  with  the 
objective  being  to  realistically  define  payoffs  of  projects  being  reviewed. 

Malehorn,  M.K.  (1986).  Embedded  training  in  naval  environments:  Current  and  future  state  (E-Tech 
TR-5938-1).  Arlington,  VA.  Eagle  Technology. 
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Provides  guidelines  for  making  early  estimates  of  training  system  requirements  and  potential 
allocation  of  a  portion  to  ET. 
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Study  Plan  for  Embedded  Training.  (1 989).  Joint  T echnical  Coordinating  Group  for  T raining  Systems  and 
Devices  (JTCG-TSD). 
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Navy,  3  Army)  systems  w'-''\i  are  currently  operational.  The  report  explores  the  manner 
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